Robotic system for picking, sorting, and placing a plurality of random and novel objects

ABSTRACT

The present disclosure generally relates to pick and place robotic systems. An exemplary system for orienting an object comprises: a scanner configured to detect a label on the object; an upper conveyor belt; a flipping conveyor belt located at an end of the upper conveyor belt, wherein the upper conveyor belt is configured to transport the object toward the flipping conveyor belt, wherein the flipping conveyor belt is configured to, in a first orientation or position, rotate and exert a frictional force on the object to reorient the object while the object is in contact with the upper conveyor belt, wherein the flipping conveyor belt is configured to, in a second orientation or position, allow the object to drop off the end of the upper conveyor belt.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application62/698,679 filed on Jul. 16, 2018, U.S. Provisional Application62/778,221 filed on Dec. 11, 2018, and U.S. Provisional Application62/827,708 filed on Apr. 1, 2019, the entire contents of which areincorporated herein by reference for all purposes.

FIELD OF THE DISCLOSURE

This relates generally to pick and place robotic systems that useartificial intelligence, computer vision, and/or mechanical systems tosort and place objects from input containers (or other receptacles) tomultiple corresponding receptacle destinations.

BACKGROUND OF THE DISCLOSURE

Many companies have inventory, distribution, or shipping systems thatrequire the sorting of a large number of objects. For example, in adistribution and fulfillment center, objects need to be collected (e.g.,batch picked) from shelves and placed together (e.g., sorted) into thecorrect boxes for shipping. As another example, in a parcel sortationcenter (e.g., of a shipping carrier), groups of parcels need to besorted to each fine-grained destination. Companies have hired humanworkers to perform such tasks. Others have also investigated usingrobotic systems to perform such tasks.

SUMMARY OF THE DISCLOSURE

Pick and place robotic systems are disclosed. An overall sorting stationsorts objects from a loading tote and places them into a set of receivercontainers or locations. When picking from a tote then placing intoanother tote, methods decide the best placing to save space and packmore objects. The end effector compliance is designed to minimize pickup time as well as minimize the risk of damage to the objects. Thesystem determines whether using a finger gripper or a suction gripper ismore effective for a particular object, and if by suction, what suctionnozzle size is suitable and whether to use high-vacuum or high-flowsuction systems. The system identifies a location for picking an objectto avoid occluding the barcode. The system identifies the optimal way tomove an object to effectively increase the chances that the barcode canbe seen in a scan station. The system decides the minimal time totransport the object between each stage of action while ensuring nounwanted collisions and stable motion.

Some embodiments described in this disclosure are directed to one ormore devices that use artificial intelligence, computer vision, and/ormechanical systems to sort and place objects from loading containers,and one or more operations related to the above that the devicesoptionally perform. The full descriptions of the embodiments areprovided in the Drawings and the Detailed Description, and it isunderstood that the Summary provided above does not limit the scope ofthe disclosure in any way.

An exemplary pick and place robotic system comprises: a robotic armhaving an end effector configured to grip objects; a sorting standadjacent the robotic arm and within working range of the robotic arm,wherein the sorting station includes a support for a containercontaining a plurality of objects; a vision system having one or moreimage sensors configured to capture image data of the container and theplurality of objects; a receptacle stand adjacent the robotic arm andwithin working range of the robotic arm, wherein the receptacle standincludes a support configured to hold a plurality of containers forreceiving one or more objects of the plurality of objects; a controlsystem in communication with the robotic arm and the vision system andhaving memory and a processor, wherein the memory includes a computerprogram having instructions executable by the processor for:identifying, based on captured image data from the vision system, alocation on the plurality of objects in the container for the endeffector to grip an object; moving the robotic arm to position the endeffector at the location; gripping the object at the location; andmoving the object from the container on the sorting stand to a containeron the receptacle stand.

In some embodiments, the computer program further includes instructionsfor: attempting to identify the object after gripping the object; and inaccordance with identifying the object, determining a container in theplurality of containers to place the object based on the identificationof the object.

In some embodiments, the computer program further includes instructionsfor: in accordance with failing to identify the object, determining agrip point and a container in the plurality of containers to place theobject based on human input.

In some embodiments, the plurality of containers are angled with respectto the ground based on a non-vertical angle of the receptacle stand withrespect to the ground.

In some embodiments, the system further comprises a scanning system foridentifying an object gripped by the robotic arm.

In some embodiments, the scanning system includes a bar code scanner.

In some embodiments, the scanning system includes one or more mirrors.

In some embodiments, identifying the location is further based on adatabase having information about potential grip location for aplurality of test objects wherein at least one object in the pluralityof objects is not in the plurality of test objects.

In some embodiments, the vision system is supported by the sortingstand.

In some embodiments, the gripper is a suction gripper.

An exemplary computer-implemented method for determining a plannedplacement and a planned orientation of a first object by a roboticsystem from a tote to a receptacle comprises: at the robotic systemcomprising a robotic arm, an end effector, an image sensor, and aprocessor: capturing image data for the first object using the imagesensor; and determining a planned placement and a planned orientation ofthe first object relative to the receptacle using the image data thatmaximize a characteristic of the receptacle and its contents.

In some embodiments, the receptacle is another tote.

In some embodiments, the characteristic of the receptacle and itscontents includes the available space for more objects after the firstobject is placed in the receptacle in accordance with a given plannedplacement and a given planned orientation.

In some embodiments, the characteristic of the receptacle and itscontents is the number of objects that can fit in the receptacle inaccordance with a given planned placement and a given plannedorientation.

In some embodiments, the method further comprises determining a plannedplacement and a planned orientation of the first object relative to thereceptacle further comprises capturing image data for a second objectdifferent from the first object using the image sensor; thecharacteristic of the receptacle and its contents is based on both: thefirst object being placed in the receptacle in accordance with a firstgiven planned placement and a first given planned orientation, and thesecond object being placed in the receptacle in accordance with a secondgiven planned placement and a second given planned orientation.

In some embodiments, the receptacle is a shelf.

In some embodiments, the characteristic of the receptacle and itscontents is whether the object will collide with the receptacle or itscurrent contents as the object is placed in accordance with a givenplanned placement and a given planned orientation.

An exemplary end effector for a pick and place robotic system comprises:a first gripper configured to grip an object with an first end of thefirst gripper, wherein: the first gripper connects to the end effectorat a second end of the first gripper, and the first end is configured tomove towards the second end in response to application of a first forceabove a threshold level of force to the first end.

In some embodiments, the first gripper is a suction gripper having atube, wherein an open end of the tube at the first end of the firstgripper is capable of providing a suction force to the object.

In some embodiments, the end effector further comprises: a trackconfigured to guide the first gripper in a direction opposite the firstforce.

In some embodiments, the threshold level of force is based on the weightof a portion of the end effector.

In some embodiments, the end effector further comprises: a springconnecting the first gripper to the end effector and configured toadjust the threshold level of force.

In some embodiments, the end effector further comprises: a secondgripper different from the first gripper.

In some embodiments, first end includes a flexible member configured tocontact the object.

In some embodiments, the flexible member is a flexible nozzle.

In some embodiments, the threshold force is dynamically adjustable.

In some embodiments, the first gripper is configured to move the firstend away from the second end when the threshold level of force is nolonger present on the first end.

In some embodiments, the end effector is connectable to a robotic arm.

An exemplary container for holding objects to be sorted in a pick andplace system comprises a receptacle having an opening for holding aplurality of objects to be sorted by the pick and place system; acompliance mechanism on a bottom side of the container configured toallow the receptacle to move downward with application of a downwardforce on the top of the container.

An exemplary sorting stand for holding a container containing objects tobe sort in a pick and place system comprises: a base for receiving thecontainer; and a compliance mechanism connected to the base andconfigured to allow the container to move downward with application of adownward force on the top of the container.

An exemplary method comprising: at a robotic system having an endeffector, wherein the end effector is configured to grip an object, anda first configuration for the end effector defines a first set ofproperties for how the end effector grips the object and a secondconfiguration for the end effector defines a second set of propertiesfor how the end effector grips the object: determining a plurality ofprobability maps of a scene including the object and at least one otherobject, each probability map corresponding respectively to a differentmotion primitive among a plurality of motion primitives, wherein: eachmotion primitive is associated with using the end effector with thefirst configuration or the second configuration to grip the object; andeach of the plurality of probability maps marks undesired regions on theobject to group the object; and choosing a motion primitive among aplurality of motion primitives to use in gripping the object based onthe plurality of probability maps; and configuring the end effectoraccording to the first configuration or second configuration associatedwith the chosen motion primitive.

In some embodiments, the plurality of motion primitives includes: afirst motion primitive using the second configuration for the endeffector; a second motion primitive different from the first motionprimitive using the second configuration for the end effector; a thirdmotion primitive using the first configuration for the end effector; anda fourth motion primitive different from the third motion primitiveusing the first configuration for the end effector.

In some embodiments, the plurality of motion primitives includes: agripping down motion primitive using the second configuration for theend effector; a flush gripping motion primitive using the secondconfiguration for the end effector; a suction down motion primitiveusing the first configuration for the end effector; a suction sidemotion primitive using the first configuration for the end effector; apushing motion primitive using either the first end configuration forthe effector or the second configuration for the end effector; atoppling motion primitive using either the first configuration for theend effector or the second configuration for the end effector; and apulling primitive using either the first configuration for the endeffector or the second configuration for the end effector.

In some embodiments, determining a plurality of probability maps of thescene including the object comprises information regarding at leastanother object.

In some embodiments, the robotic system has not determined previously aprobability map of a scene including the object.

In some embodiments, the plurality of probability maps are pixel-wiseprobability maps.

In some embodiments, the plurality of probability maps are pixel-wisebinary probability maps.

In some embodiments, determining a probability map of the scenecorresponding to a motion primitive associated with the firstconfiguration for the end effector further includes: determining, usinga machine learning algorithm, a proposed suction point corresponding toa pixel of an image of the scene, a local surface geometry of theproposed suction point, and a probability of picking the object at theproposed suction point; and outputting a pixel-wise binary probabilitymap of the scene.

In some embodiments, the first configuration for the end effector isdefines a first attachment to be attached to the end effector; anddetermining a plurality of probability maps of a scene further includes:determining a first probability map of the scene corresponding to amotion primitive associated with the end effector coupled with the firstattachment.

In some embodiments, the robotic system further comprises a firstsuction generator and a second suction generator different from thefirst suction generator; and choosing a motion primitive among aplurality of motion primitives to use in picking the object based on theplurality of probability maps further comprises: in accordance with thechosen motion primitive being associated with using the firstconfiguration for the end effector, determining whether to generatesuction using the first suction generator or the second suctiongenerator.

In some embodiments, the robotic system further comprises a first sensormeasuring a first property associated with the first suction generator,and a second sensor measuring a second property different from the firstproperty associated with the second suction generator, the methodfurther comprising: in accordance with determining to generate suctionusing the first suction generator, determining a first suction gripbased on the first property measured at the first sensor; and inaccordance with determining to generate suction using the second suctiongenerator, determining a second suction grip based on the secondproperty measured at the second sensor.

In some embodiments, determining a probability map of the scenecorresponding to a motion primitive associated with the secondconfiguration for the end effector further includes: determining aproposed three-dimensional grip location corresponding to athree-dimensional representation of the scene, a middle point between afirst finger and a second finger of the end effector configuredaccording to the second configuration; an angle corresponding to theorientation of the first finger and the second finger; a width betweenthe first finger and the second finger at the proposed grip location; aprobability of picking the object at the proposed three-dimensionallocation; and outputting a pixel-wise binary probability map of thescene.

In some embodiments, the method further comprises determining thedistance between the proposed three-dimensional grip location relativeto a side of a receptacle containing the object and the at least oneother object; and determining whether to use the third motion primitiveor the fourth motion primitive based on the plurality of probabilitymaps, the width between the first finger and the second finger at theproposed grip location, and the distance between the proposedthree-dimensional grip location relative to a side of a receptaclecontaining the object and the at least one other object.

An exemplary computer-implemented method for determining a location in ascene, wherein the scene includes a plurality of objects to be picked,for a robotic system to acquire an object in the plurality of objectscomprises: at the robotic system comprising a robotic arm, an endeffector, an image sensor, a database, and a processor: capturing imagedata for the scene using the image sensor; generating, based on theimage data for the scene, a probability map comprising a plurality ofprobabilities each corresponding to a region in a plurality of regionson the object, wherein the plurality of probabilities is based on: thelikelihood that the corresponding region in the plurality of regions isa barcode portion; and data stored in the database; and determining alocation on the object by selecting a region on the object correspondingto a probability in the plurality of probabilities that exceeds athreshold probability.

In some embodiments, the method further comprises in accordance with adetermination that the end effector coming into contact with the objectat a region causes the barcode on the object to be occluded, aprobability of zero is assigned to the region.

In some embodiments, the method further comprises determining aplurality of locations on the object by selecting regions on the objectcorresponding to probabilities in the plurality of probabilities thatexceeds a threshold probability, wherein the distance between thelocations is beyond a threshold distance; and attempting to acquire theobject at each of the plurality of locations on the object until eitherat least one of them is successful or none are successful in acquiringthe object.

In some embodiments, the method further comprises in accordance with adetermination that no region has corresponding likelihood of being abarcode region over a threshold likelihood or if none of theprobabilities are above a troubleshooting threshold, entering atroubleshooting mode.

In some embodiments, the troubleshooting mode comprises: acquiring theobject; rotating, pushing, toppling, or pulling the object; and puttingthe object back into the scene.

In some embodiments, the troubleshooting mode comprises: attempting toacquire the object at the location; and in accordance with being unableto acquire the object at the location, setting the probabilitiescorresponding to regions within a threshold radius of the location tozero.

An exemplary computer-implemented method for scanning a barcode on anobject using a robotic system comprises: at the robotic systemcomprising a robotic arm, a device gripper, and an image sensor:gripping the object using the gripper; estimating the location of abarcode on the object; determining a planned movement of the object,wherein: the planned movement comprises translation and rotation; andthe planned movement is based on the location of the image sensorrelative to the estimated location of the barcode on the object; movingthe object in accordance with the planned movement; capturing image datafor the object using the image sensor; identifying a barcode on theobject using the image data; and scanning the barcode on the object.

In some embodiments, the robotic system further comprises a plurality ofbar code scanners, each aligned at different angles and orientations.

In some embodiments, the robotic system further comprises one or moremirrors configured so that an image of the object and its surroundingsincludes the reflection of the one or more mirrors, wherein reflectionsinclude surfaces of the object not directly visible from the imagesensor.

In some embodiments, the mirrors are placed in a manner to optimize theprobability that the bar code is visible in a single image shot thatincludes the object and reflections of the object in the mirrors.

In some embodiments, the planned movement is set to minimize the time tocomplete the planned movement subject to a constraint.

In some embodiments, the constraint is that the probability of theobject falling off of the robotic arm is below a threshold.

In some embodiments, the method further comprises in accordance with novalid barcode being detected after scanning or redundant barcodedetection in the event of double picking, entering an abort and repickor a troubleshooting mode.

In some embodiments, the troubleshooting mode comprises: rotating ortranslating the object; putting the object back into the scene; andgripping the object.

In some embodiments, the troubleshooting mode comprises alerting a userfor assistance.

In some embodiments, the robotic system further comprises externallighting configured to shine light onto the object, and thecomputer-implemented method further comprises intelligently controllingthe external lighting to improve the visibility of the barcode on theobject.

In some embodiments, minimizing the time to complete the plannedmovement subject to a constraint is set by a machine learning algorithm.

In some embodiments, the system further comprises a checking system forconfirming the correctness of the contents of one or more of theplurality of containers.

In some embodiments, the checking system comprises a plurality ofdisplays each corresponding to a container in the plurality ofcontainers and indicating a count of the number of objects in itscorresponding container as tracked by the robotic system.

In some embodiments, the checking system comprises: a plurality ofbarcodes each corresponding to a container in the plurality ofcontainers, wherein each barcode corresponds to information regardingthe contents of the corresponding container as tracked by the roboticsystem, and a screen configured to display the information regarding thecontents of a container as tracked by the robotic system in accordancewith a user scanning the barcode corresponding to the container.

In some embodiments, the support is configured to angle the plurality ofcontainers with respect to the ground.

In some embodiments, the support includes a plurality of chutes todirect an object of the plurality of objects into a container of theplurality of containers.

In some embodiments, the system further comprises: a funnel above thesorting stand configured to redirect objects dropped from above into thecontainer.

In some embodiments, the system further comprises: a container conveyorconfigured to transport containers from an input location to a sortinglocation.

In some embodiments, the container conveyor includes a justifyingconveyor that positions an input container adjacent to the robotic armfor sorting.

In some embodiments, the container conveyor includes an input conveyorfor loading an input container.

In some embodiments, the container conveyor includes an output conveyorfor carrying away a sorted container.

In some embodiments, the plurality of chutes are angles towards theplurality of containers.

In some embodiments, the plurality of chutes includes a plurality ofoptical detectors to determine when an object is placed into a containerassociated with a chute.

In some embodiments, the receptacle stand is removeably coupled to thesorting stand.

In some embodiments, the receptacle stand includes wheels.

In some embodiments, the first gripper is a suction gripper having arigid structure that contacts an object to be sorted.

In some embodiments, the first gripper is a suction gripper configuredto grip a plastic bag associated with the object to be sorted.

In some embodiments, the end effector is mounted on a phalange of arobotic arm via a magnetic connector.

In some embodiments, the magnetic connector is configured to disengagein response to a threshold level of force being applied to the endeffector.

In some embodiments, the magnetic connector provides for electricalconnections to the phalange.

In some embodiments, the magnetic connector provides for a suctionconnection to the phalange.

An exemplary system for coupling a detachable tool to a motion devicecomprises: a first magnetic ring affixed to a distal end of the motiondevice, wherein the motion device and the first magnetic ring form afirst hollow chamber extending through a length of the motion device andthrough a center of the first magnetic ring; a second magnetic ringaffixed to a proximal end of the detachable tool, wherein the secondmagnetic ring and the detachable tool forms a second hollow chamberextending from a center of the second magnetic ring and the detachabletool, wherein the first magnetic ring and the second magnetic ring areconfigured to automatically couple together via a magnetic field in analigned manner, and wherein the coupling of the first magnetic ring andthe second magnetic ring joins the first hollow chamber and the secondhollow chamber to allow for a pass-through mechanism.

In some embodiments, the pass-through mechanism is a vacuumpass-through.

In some embodiments, the pass-through mechanism is an electronicpass-through.

In some embodiments, the pass-through mechanism is a mechanicalpass-through.

In some embodiments, the detachable tool comprises a groove configuredto interface with at least one slot on a tool rack.

In some embodiments, the at least one slot has an opening that is widerat the beginning than at the end to facilitate interfacing with thegroove.

In some embodiments, the tool rack is ferrous.

In some embodiments, the system further comprises a tool changer base atthe distal end of the motion device, wherein the tool changer base formspart of the first hollow chamber.

In some embodiments, a cross-section of the first magnetic ring and across-section of the second magnetic ring are identical.

In some embodiments, the ring is an ellipse shape.

In some embodiments, the ring is a circle shape.

In some embodiments, the ring is a polygon shape.

In some embodiments, the detachable tool has a suction cup at the distalend.

In some embodiments, the detachable tool has an electrically orpneumatically activated gripper at the distal end.

An method for decoupling a detachable tool from a motion device,comprises: causing the motion device to move along a first directiontoward a slot of a tool rack while the detachable tool is coupled to adistal end of the motion device, wherein the motion device comprises afirst magnetic ring and the detachable tool comprises a second magneticring, and wherein the first magnetic ring and the second magnetic ringare configured to couple automatically via a magnetic field in analigned manner; causing the motion device to align a groove of thedetachable tool with the slot of the tool rack; causing the motiondevice to move away from the tool rack in along a second direction todecouple the detachable tool from the distal end of the motion device,wherein the slot of the tool rack is configured to retain the detachabletool.

In some embodiments, the detachable tool is a first detachable tool, themethod further comprises positioning the motion device in proximity to asecond detachable tool held in a second slot of the tool rack to couplethe distal end of the motion device with a proximal end of the seconddetachable tool; and moving, using the motion device, the detachabletool along the first direction out of the second slot of the tool rack.

In some embodiments, the method further comprises storing a location ofthe detachable tool in a memory of a computer.

In some embodiments, the method further comprises: causing a distal endof the detachable tool to grip an object; causing the motion device tomove the object; causing the distal end of the detachable tool torelease the object.

In some embodiments, the first direction is along a horizontal axis.

In some embodiments, the second direction is along a vertical axis.

An exemplary apparatus for vacuum-gripping a deformable bag, comprises:a primary chamber, wherein a proximal end of the primary chamber isconnected to an air flow source, and wherein the primary chamber isconfigured to, upon an activation of the air flow source, receive aportion of the deformable bag via a distal end of the primary chamber; asecondary chamber surrounding the primary chamber, wherein the secondarychamber is connected to the primary chamber via a plurality ofconnections to allow for air passage, and wherein the activation of theair flow source causes a lateral wall of the primary chamber to grip theportion of the deformable bag via pressure differential between aninside of the deformable bag and the secondary chamber.

In some embodiments, the primary and secondary chambers are nested.

In some embodiments, a subset of the plurality of connections arearranged radially on the lateral wall of the primary chamber.

In some embodiments, the air flow source is a vacuum source.

In some embodiments, the proximal end of the apparatus comprises a firstmagnetic ring configured to automatically couple with a second magneticring of a motion device in an aligned manner.

In some embodiments, the distal end of the apparatus comprises a suctioncup configured to grip rigid surfaces.

In some embodiments, the apparatus further comprises a first hollowcylinder, wherein the first follow cylinder forms the primary chamber.

In some embodiments, the apparatus further comprises a second hollowcylinder, wherein the first hollow cylinder is placed inside the secondhollow cylinder, and wherein a space between the first hollow cylinderand the second hollow cylinder forms the secondary chamber.

In some embodiments, the plurality of connections are formed via aplurality of holes on a lateral wall of the first follow cylinder.

In some embodiments, the first hollow cylinder is made of plastic,metal, or a combination thereof.

In some embodiments, the second hollow cylinder is made of plastic,metal, or a combination thereof.

In some embodiments, the activation of the air flow source comprisesactivation of a vacuum pass-through.

In some embodiments, the apparatus further comprises one or moreprocessors; memory; and one or more programs, wherein the one or moreprograms are stored in the memory and configured to be executed by theone or more processors, the one or more programs comprises instructionsfor: identifying a region on the deformable bag; identifying a distancebetween the primary chamber and the region on the deformable bag, basedon the distance, determining whether to activate the flow source.

In some embodiments, the one or more programs further compriseinstructions for deactivating the flow source.

An exemplary system for orienting an object comprises: a scannerconfigured to detect a label on the object; an upper conveyor belt; aflipping conveyor belt located at an end of the upper conveyor belt,wherein the upper conveyor belt is configured to transport the objecttoward the flipping conveyor belt, wherein the flipping conveyor belt isconfigured to, in a first orientation, rotate and exert a frictionalforce on the object to reorient the object while the object is incontact with the upper conveyor belt, wherein the flipping conveyor beltis configured to, in a second orientation, allow the object to drop offthe end of the upper conveyor belt, and wherein the flipping conveyorbelt is configured to move from the first orientation to the secondorientation based on an output of the scanner.

In some embodiments, the end of the upper conveyor belt is a first end,the system further comprises a curved chute at a second end of the upperconveyor belt, wherein the upper conveyor belt is configured totransport the object toward the curved chute, and wherein the curvedchute is configured to rotate the object by 180 degrees.

In some embodiments, the system further comprises a lower conveyor beltconfigured to receive the object after it is rotated by the curvedchute.

In some embodiments, the flipping conveyor belt is angled to the upperconveyor belt in the first orientation.

In some embodiments, the upper conveyor belt is cleated.

In some embodiments, the flipping conveyor belt is configured to pull aportion of the package upwards and away from upper conveyor belt in thefirst orientation.

In some embodiments, the flipping conveyor belt is configured to swingfrom the first orientation to the second orientation.

In some embodiments, the system further comprises a scanner configuredto scan one or more surfaces of the object.

In some embodiments, the object is a deformable bag.

In some embodiments, the object is a box.

An exemplary method for orienting an object, the method comprises:causing an upper conveyor belt to move the object toward a flippingconveyor belt located at an end of the upper conveyor belt; determining,based on an output from a scanner, whether the object is in one of oneor more predefined orientations; in accordance with a determination thatthe object is not in in one of one or more predefined orientations,causing the flipping conveyor belt and the upper conveyor belt to runsimultaneously to reorient the object while the flipping conveyor beltis in a first orientation; in accordance with a determination that theobject is in one of one or more predefined orientations, causing theflipping conveyor belt to move to a second orientation such that theobject is dropped off the end of the upper conveyor belt.

In some embodiments, the end of the upper conveyor belt is a first end,the method further comprises: in accordance with a determination thatthe object is not in one of one or more predefined orientations,determining, based on an output from an optical sensor, a height of theobject; in accordance with a determination that the height is below acertain threshold, causing the upper conveyor belt to move the objecttoward a second end; in accordance with a determination that the heightis above a certain threshold, forgoing causing the upper conveyor beltto move the object toward the second end.

In some embodiments, the upper conveyor belt is cleated.

In some embodiments, the flipping conveyor belt is configured to pull aportion of the package upwards and away from upper conveyor belt.

In some embodiments, determining whether the object is in one of one ormore predefined orientations is based on a configuration of a downstreamsorter.

In some embodiments, determining whether the object is in one of one ormore predefined orientations comprises: scanning, using the scanner, asurface of the object to obtain image data; determining, based on theimage data, whether the image data includes information related to theobject.

In some embodiments, the method further comprises: in accordance with adetermination that the image data includes information related to theobject, determining that the object is in one of one or more predefinedorientations; and in accordance with a determination that the image datadoes not include information related to the object, determining that theobject is not in one of one or more predefined orientations.

In some embodiments, the information related to the object comprises abarcode.

In some embodiments, the method further comprises: in accordance with adetermination that the object is not in in one of one or more predefinedorientations, determining whether a height of the object exceeds athreshold; in accordance with a determination that the height of theobject exceeds the threshold, causing the flipping conveyor belt and theupper conveyor belt to run simultaneously to reorient the object whilethe flipping conveyor belt is in a first orientation; in accordance witha determination that the height of the object does not the threshold,reversing movement of the upper conveyor belt to transport the object toa curved chute.

An exemplary method for orienting an object comprises causing an upperconveyor belt to move the object toward a flipping conveyor belt locatedat an end of the upper conveyor belt; determining, based on an outputfrom a scanner, whether a code on the package is read; in accordancewith a determination that the code on the package is read, causing theflipping conveyor belt and the upper conveyor belt to run simultaneouslyto reorient the object while the flipping conveyor belt is in a firstorientation; in accordance with a determination that the code on thepackage is not read, causing the flipping conveyor belt to move to asecond orientation such that the object is dropped off the end of theupper conveyor belt.

An exemplary compliant mechanism, comprises: a motion device, wherein adistal surface of the motion device comprises a first plurality ofmagnetic components; an end effector, wherein the end effectorcomprises: a second plurality of magnetic components arranged on aproximal surface of the end effector in a same configuration as thefirst plurality of magnetic components, a rod, and an elongated memberextending through a hole in the proximal surface of the end effector,wherein: a proximal end of the elongated member is affixed to the distalsurface of the motion device, and the elongated member comprises an endstopper piece configured to prevent a distal end of the elongated memberfrom passing through the hole in the proximal surface of the endeffector.

In some embodiments, the compliant mechanism is configured to: while theproximal surface of the end effector is attached to the distal surfaceof the motion device via the first plurality of magnetic components andthe second plurality of magnetic components, in response to receiving alateral force to the rod, cause one or more of the first plurality ofmagnetic components to detach from one or more of the second pluralityof magnetic components, and in response to stopping receiving thelateral force, cause the proximal surface of the end effector toautomatically attach to the distal surface of the motion device via thefirst plurality of magnetic components and the second plurality ofmagnetic components.

In some embodiments, one of the distal surface of the motion device andthe proximal surface of the end effector comprises one or more pins; andthe other one of the distal surface of the motion device and theproximal surface of the end effector comprises one or more openings forreceiving the one or more pins.

In some embodiments, the one or more pins each comprises a tapered top.

In some embodiments, the motion device comprises at least a portion of arobotic arm.

In some embodiments, the second plurality of magnetic components arespaced circumferentially on the proximal surface of the end effector.

In some embodiments, the second plurality of magnetic components areaffixed to the proximal surface of the end effector via a screwmechanism.

In some embodiments, the elongated member comprises a screw.

In some embodiments, the proximal end of the elongated member is affixedto the distal surface of the motion device via a screw mechanism.

In some embodiments, the end stopper piece comprises a bolt.

In some embodiments, the proximal end of the rod is attachable to aflexible tube in the motion device.

In some embodiments, the elongated member is a first elongated member,the compliant mechanism further comprises a second elongated member.

In some embodiments, the mechanism further comprises one or more sensorsfor detecting detachment between a portion of the proximal surface ofthe end effector and a portion of the distal surface of the motiondevice.

An exemplary end effector attachable to a motion device, comprises: aplurality of magnetic components arranged on a proximal surface of theend effector; a rod, and an elongated member extending through a hole inthe proximal surface of the end effector, wherein: a proximal end of theelongated member is attachable to a distal surface of the motion device,and the elongated member comprises an end stopper piece configured toprevent a distal end of the elongated member from passing through thehole in the proximal surface of the end effector.

In some embodiments, the plurality of magnetic components is a secondplurality of magnetic components, the second plurality of magneticcomponents are arranged in a same configuration as a first plurality ofmagnetic components arranged on the distal surface of the motion device,and the end effector is configured to: while the proximal surface of theend effector is attached to a distal surface of the motion device viathe first plurality of magnetic components and the second plurality ofmagnetic components, in response to receiving a lateral force to therod, cause one or more of the first plurality of magnetic components todetach from one or more of the second plurality of magnetic components,and in response to stopping receiving the lateral force, cause theproximal surface of the end effector to automatically attach to thedistal surface of the motion device via the first plurality of magneticcomponents and the second plurality of magnetic components.

In some embodiments, the end effector further comprises one or more pinson the proximal surface of the end effector.

In some embodiments, the one or more pins each comprises a tapered top.

In some embodiments, the plurality of magnetic components are spacedcircumferentially on the proximal surface of the end effector.

In some embodiments, the plurality of magnetic components are affixed tothe proximal surface of the end effector via a screw mechanism.

In some embodiments, the elongated member comprises a screw.

In some embodiments, the end stopper piece comprises a bolt.

An exemplary compliance mechanism comprises: a motion device; an endeffector coupled to the motion device, wherein the end effectorcomprises: a sheath structure, wherein the sheath structure comprises aslot, and wherein the sheath structure is configured to remainstationary relative to the motion device when the end effector iscoupled to the motion device; a rod, wherein a portion of the rod isenclosed by the sheath structure; a protruded piece affixed to the rod,wherein the protruded piece is positioned within the slot of the sheath;wherein the compliance mechanism is configured to: when the distal endof the rod is in not contact with an object, causing the distal end ofthe rod to move responsive to movement of the motion device, and whenthe distal end of the rod is in contact with an object and the motiondevice moves toward the object, causing the distal end of the rod toremain stationary by causing the sheath structure to move along alongitudinal direction of the rod.

In some embodiments, the sheath structure is configured to start movingalong the rod when a resistance force between the distal end of the endeffector and the object is above a predefined threshold.

In some embodiments, the slot of the sheath is configured to slide alongthe protruded piece when the sheath structure moves along thelongitudinal direction of the rod.

In some embodiments, the protruded piece is of a round shape.

In some embodiments, the protruded piece is of a polygon shape.

In some embodiments, the protruded piece is affixed to a ring that wrapsaround the rod.

In some embodiments, the end effector comprises a casing enclosing thesheath structure.

In some embodiments, the end effector comprises a first end stopper anda second end stopper on the rod; and wherein the casing is between thefirst end stopper and the second end stopper.

In some embodiments, the motion device comprises at least a portion of arobotic arm.

In some embodiments, the end effector is coupled to a phalange of therobotic arm.

In some embodiments, the end effector is coupled to the motion devicevia one or more magnetic components on the end effector.

In some embodiments, the distal end of the end effector comprises agripper.

In some embodiments, the gripper comprises a suction cup.

In some embodiments, the rod is configured to accommodate vacuumpass-through.

In some embodiments, the mechanism further comprises one or more sensorsfor detecting a movement of the sheath structure along the rod.

In some embodiments, an exemplary gripping apparatus, comprises: asheath structure, wherein the sheath structure comprises a slot, andwherein the sheath structure is configured to remain stationary relativeto a motion device when the gripping apparatus is attached to the motiondevice; a rod, wherein a portion of the rod is enclosed by the sheathstructure; a protruded piece affixed to the rod, wherein the protrudedpiece is positioned within the slot of the sheath; wherein the grippingapparatus is configured to: when the distal end of the rod is in notcontact with an object, causing the distal end of the rod to moveresponsive to movement of the motion device, and when the distal end ofthe rod is in contact with an object and the motion device moves towardthe object, causing the distal end of the rod to remain stationary bycausing the sheath structure to move along a longitudinal direction ofthe rod.

In some embodiments, the slot of the sheath is configured to slide alongthe protruded piece when the sheath structure moves along thelongitudinal direction of the rod.

In some embodiments, the gripping apparatus further comprises a casingenclosing the sheath structure.

In some embodiments, the gripping apparatus further comprises a firstend stopper and a second end stopper on the rod; and wherein the casingis between the first end stopper and the second end stopper.

In some embodiments, the motion device comprises at least a portion of arobotic arm.

In some embodiments, the distal end of the gripping apparatus comprisesa gripper.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

For a better understanding of the various described embodiments,reference should be made to the Detailed Description below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 illustrates an exemplary pick and place system in accordance withsome embodiments.

FIG. 2A illustrates a pick and place system sorting objects inaccordance with some embodiments.

FIG. 2B illustrates a pick and place system sorting objects inaccordance with some embodiments.

FIG. 2C illustrates a pick and place system sorting objects inaccordance with some embodiments.

FIG. 3 illustrates a pick and place system sorting objects in accordancewith some embodiments.

FIG. 4A illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4B illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4C illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4D illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4E illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4F illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4G illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4H illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4I illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4J illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 4K illustrates exemplary placements of objects in accordance withsome embodiments.

FIG. 5A is a flow diagram illustrating exemplary methods of placingobjects in accordance with some embodiments.

FIG. 5B is a flow diagram illustrating exemplary methods of placingobjects in accordance with some embodiments.

FIG. 5C is a flow diagram illustrating exemplary methods of placingobjects in accordance with some embodiments.

FIG. 6A illustrates an exemplary compliant end effector in accordancewith some embodiments.

FIG. 6B illustrates an exemplary compliant end effector in accordancewith some embodiments.

FIG. 6C illustrates an exemplary compliant end effector in accordancewith some embodiments.

FIG. 6D illustrates an exemplary compliant end effector in accordancewith some embodiments.

FIG. 6E illustrates an exemplary compliant end effector in accordancewith some embodiments.

FIG. 6F illustrates an exemplary compliant end effector in accordancewith some embodiments.

FIG. 6G illustrates an exemplary compliant end effector in accordancewith some embodiments.

FIG. 7 illustrates an exemplary system for switching end effectorsbetween a gripper and various suction nozzles as well as betweenhigh-vacuum and high-flow suction systems in accordance with someembodiments.

FIG. 8 illustrates an exemplary system for switching end effectorsbetween a gripper and various suction nozzles as well as betweenhigh-vacuum and high-flow suction systems in accordance with someembodiments.

FIG. 9 illustrates an exemplary system for switching end effectorsbetween a gripper and various suction nozzles as well as betweenhigh-vacuum and high-flow suction systems in accordance with someembodiments.

FIG. 10A illustrates an exemplary system for detecting barcodes in orderto avoid picking on them in accordance with some embodiments.

FIG. 10B illustrates an exemplary system for detecting barcodes in orderto avoid picking on them in accordance with some embodiments.

FIG. 10C illustrates an exemplary system for detecting barcodes in orderto avoid picking on them in accordance with some embodiments.

FIG. 11A is a flow diagram illustrating exemplary methods of pickingbased on barcode scanning in accordance with some embodiments.

FIG. 11B is a flow diagram illustrating exemplary methods of pickingbased on barcode scanning in accordance with some embodiments.

FIG. 11C is a flow diagram illustrating exemplary methods of pickingbased on barcode scanning in accordance with some embodiments.

FIG. 12 illustrates exemplary systems for barcode scanning in accordancewith some embodiments.

FIG. 13A is a flow diagram illustrating exemplary methods of barcodescanning in accordance with some embodiments.

FIG. 13B is a flow diagram illustrating exemplary methods of barcodescanning in accordance with some embodiments.

FIG. 13C is a flow diagram illustrating exemplary methods of barcodescanning in accordance with some embodiments.

FIG. 13D is a flow diagram illustrating exemplary methods of barcodescanning in accordance with some embodiments.

FIG. 14A depicts an exemplary pick and place system in accordance withsome embodiments.

FIG. 14B depicts an exemplary pick and place system in accordance withsome embodiments.

FIG. 15 depicts an exemplary put wall in accordance with someembodiments.

FIG. 16 depicts an exemplary mobile put wall coupled to a conveyor inaccordance with some embodiments.

FIG. 17 depicts an exemplary mobile put wall decoupled from a conveyorin accordance with some embodiments.

FIG. 18 depicts an exemplary user interface for a user to interact witha pick and place system.

FIG. 19A depicts an exemplary magnetic coupling for an end effector.

FIG. 19B depicts an exemplary magnetic coupling for an end effector.

FIG. 19C depicts an exemplary magnetic coupling for an end effector anda cross-section view of the magnetic coupling.

FIG. 19D depicts an exemplary magnetic coupling for an end effector.

FIG. 19E depicts an exemplary magnetic coupling for a detachable tool inaccordance with some embodiments.

FIG. 19F depicts an exemplary magnetic coupling for a detachable tool inaccordance with some embodiments.

FIG. 19G depicts an exemplary magnetic coupling for a detachable tool inaccordance with some embodiments.

FIG. 19H depicts an exemplary tool rack for a detachable tool inaccordance with some embodiments.

FIG. 19I depicts an exemplary tool rack for a detachable tool inaccordance with some embodiments.

FIG. 19J depicts an exemplary tool rack for a detachable tool inaccordance with some embodiments.

FIG. 20 depicts an exemplary loose bag cup gripper.

FIG. 21A depicts an exemplary loose bag cup gripper in accordance withsome embodiments.

FIG. 21B depicts an exemplary loose bag cup gripper in accordance withsome embodiments.

FIG. 21C depicts an exemplary loose bag cup gripper in accordance withsome embodiments.

FIG. 21D depicts an exemplary loose bag cup gripper in accordance withsome embodiments.

FIG. 21E depicts an exemplary loose bag cup gripper in accordance withsome embodiments.

FIG. 22A depicts an exemplary loose bag cup gripper in accordance withsome embodiments.

FIG. 22B depicts an exemplary loose bag cup gripper in accordance withsome embodiments.

FIG. 23 depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 24A depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 24B depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 24C depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 24D depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 24E depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 24F depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 25A depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 25B depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 25C depicts an exemplary package orientation system in accordancewith some embodiments.

FIG. 25D depicts an exemplary package orientation system in accordancewith some embodiments.

DETAILED DESCRIPTION

The detailed description set forth below in connection with the appendeddrawings is intended as a description of various configurations and isnot intended to represent the only configurations in which the conceptsdescribed herein can be practiced. The detailed description includesspecific details for the purpose of providing a thorough understandingof various concepts. However, it will be apparent to those skilled inthe art that these concepts can be practiced without these specificdetails. In some instances, well-known structures and components areshown in block diagram form in order to avoid obscuring such concepts.

Examples of systems and methods for picking, sorting, and placing aplurality of random and novel objects will now be presented withreference to various electronic and mechanical devices and methods.These devices and methods will be described in the following detaileddescription and illustrated in the accompanying drawing by variousblocks, components, circuits, steps, processes, algorithms, etc.(collectively referred to as “elements”). These elements can beimplemented using electronic hardware, computer software, or anycombination thereof. Whether such elements are implemented as hardwareor software depends upon the particular application and designconstraints imposed on the overall system.

By way of example, an element, or any portion of an element, or anycombination of elements of the various electronic systems can beimplemented using one or more processors. Examples of processors includemicroprocessors, microcontrollers, graphics processing units (GPUs),central processing units (CPUs), application processors, digital signalprocessors (DSPs), reduced instruction set computing (RISC) processors,systems on a chip (SoC), baseband processors, field programmable gatearrays (FPGAs), programmable logic devices (PLDs), state machines, gatedlogic, discrete hardware circuits, and other suitable hardwareconfigured to perform the various functionalities described throughoutthis disclosure. One or more processors in the processing system canexecute software. Software shall be construed broadly to meaninstructions, instruction sets, code, code segments, program code,programs, subprograms, software components, applications, softwareapplications, software packages, routines, subroutines, objects,executables, threads of execution, procedures, functions, etc., whetherreferred to as software, firmware, middleware, microcode, hardwaredescription language, or otherwise.

Accordingly, in one or more examples, the functions described for thesystem for picking, sorting, and placing can be implemented in hardware,software, or any combination thereof. If implemented in software, thefunctions can be stored on or encoded as one or more instructions orcode on a computer-readable medium. Computer-readable media can includetransitory or non-transitory computer storage media for carrying orhaving computer-executable instructions or data structures storedthereon. Both transitory and non-transitory storage media can be anyavailable media that can be accessed by a computer as part of theprocessing system. By way of example, and not limitation, suchcomputer-readable media can include a random-access memory (RAM), aread-only memory (ROM), an electrically erasable programmable ROM(EEPROM), optical disk storage, magnetic disk storage, other magneticstorage devices, combinations of the aforementioned types ofcomputer-readable media, or any other medium that can be used to storecomputer-executable code in the form of instructions or data structuresaccessible by a computer. Further, when information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or combination thereof) to a computer, the computeror processing system properly determines the connection as a transitoryor non-transitory computer-readable medium, depending on the particularmedium. Thus, any such connection is properly termed a computer-readablemedium. Combinations of the above should also be included within thescope of the computer-readable media. Non-transitory computer-readablemedia exclude signals per se and the air interface.

FIG. 1 illustrates an exemplary pick and place system 100 according tosome embodiments of the present technology includes robotic arm 102,sorting stand 150, and receptacle stand 180. In some cases, sortingstand 150 and/or receptacle stand 180 are replaced by or include a toteconveyor and/or put wall, respectively, similar to that described withrespect to FIGS. 14-17.

In FIG. 1, robotic arm 102 grips objects from tote 152 in sorting stand150, identifies the gripped objects, and places the gripped objects atlocations in receptacle stand 180 (e.g., bins 182). Pick and placesystem 100 also includes a control system (not shown) that includes aprocessor, memory, communications interfaces, and other components. Pickand place system 100 is configured to pick and place a wide variety ofobjects including novel objects that the system has not previouslygripped, placed, or even seen.

Robotic arm 102 includes base 104 for mounting to a support surface(e.g., the floor or some other support structure). Frame 106 isrotatably connected to base 104. Lower arm 108 is rotatably connected toframe 110. Upper arm 112 is rotatably connected to lower arm 108. Endeffector 114 (FIG. 2A) is rotatably connected to upper arm 112. Endeffector 114 includes one or more grippers. In the case of FIG. 1,gripper 116 is a suction gripper. Other grippers, such as grippingfingers or other type of suction grippers (e.g., FIGS. 6A, 19A, and 20),can also be used. In some cases, end effector 114 is compliant (see FIG.6A and FIGS. 19A-D) and/or multi-purpose. The control system providesinstructions and/or command signals for moving (e.g., rotating,extending, retracting) the various components of robotic arm 102.

Sorting stand 150 includes support structure 154, which is a system ofmetal support members bolted together. The side of support structure 154opposite robotic arm 102 includes an opening allowing a tote (e.g., tote152) or other receptacle to be inserted into sorting stand 150. Sorting150 optionally includes base 156 for supporting receptacles. Sortingstand 150 also includes a vision system with four cameras 158, eachhaving one or more image sensors (e.g., visible light and/or infraredsensors). The vision system can have any number of cameras and belocated in other locations or supported by other structures. In somecases, cameras 158 capture image data that includes visible light data(e.g., RGB data) and/or depth information (e.g., how far objects in theimage are from the camera). The captured image data is sent to thecontrol system for processing.

Receptacle stand 180 includes support structure 184 for bins 182 orother receptacles that hold objects that are sorted by pick and placesystem 100. Support structure 184 is optionally angled (e.g., isnon-vertical) to reduce the probability of objects falling out of bins182. Additionally, the angle can be chosen so that robotic arm 102 doesnot need to hold a gripped object out horizontally prior to placing itin one of the bins. Receptacle stand 180 is positioned adjacent tosorting stand 150 so that both stands are within working range ofrobotic arm 102 and with enough room apart that a human can work betweenthe components of pick and place system 100 (e.g., so a human can handleany objects that cannot be identified or that fall off the bins).

FIGS. 2A-3 depict pick and place system 100 operating according to someembodiments of the present technology. FIG. 2A depicts pick and placesystem 100 having two bins 152A and 152B. Bin 152A includes objects 170for sorting. Using the vision system, the control system determines anobject to sort or a location that has a high probability of being asuitable grasp spot, as described further below. FIG. 2B depicts pickand place system 100 after the control system determined that a locationon object 170A was the next place for a grip attempt. The control systemmoves robotic arm 102 so that end effector 114 contacts the location onobject 170A. FIG. 2C depicts pick and place system 100 after the controlsystem instructed robotic arm 102 to lift gripped object 170A and movethe object towards bin 152B. The control system can then identify theobject (e.g., by scanning a bar code or using image recognition oranalyzing other properties of the object) and determine where the objectshould be placed. FIG. 3 depicts pick and place system 100 placingobject 170A into bin 182A. For example, the control system optionallydetermines based on the object identification that the object should beplaced in bin 182A. Note that robotic arm 102 need not hold object 170Aout completely horizontally because support structure 184 is angled. Theholding angle is optimized to balance the gripping force and placementmotion constraint. An overall sorting station where a robotic systemsorts objects from a loading tote and places them into a set of receiverboxes. When the robotic system cannot recognize an object, such as thebar code is unreadable, it notifies the human assistant, who can thencomplete the task. The receiver boxes are placed in positions tominimize the time it takes to move objects, to ensure that objects donot fall out, and so that the human assistant can reach them.

In one example, in the case of the system not being able to scan theobject, the system drops it on a slope and it will return to the initialtote from which the robot is picking objects. Optionally, a designatedarea of the sorting stand (e.g., a bottom shelf) has a slope that allowsan object placed there to be recycled back to the tote. A recyclingmechanism is present to enable another trial. In some embodiments, thesystem redirects to a human station or requests human input orintervention in the case of repeated failures.

In some embodiments, the pick and place system including a checkingsystem for confirming the correctness of the contents of the receptaclesafter some objects have been placed in them. For example, eachreceptacle may include a display (e.g. LCD screen, LED digital counter)that indicates a count of the number of objects in its correspondingreceptacle as tracked by the robotic system. In other embodiments, eachreceptacle may include a barcode, which corresponds to informationregarding the contents of the corresponding receptacle as tracked by therobotic system. When a user (e.g. packaging personnel) scans thebarcode, a screen displays the information regarding the contents of thecorresponding receptacle as tracked by the robotic system. The user canthen verify the displayed information against what he/she sees in thereceptacle.

FIGS. 4A-4K illustrate exemplary placements of objects in accordancewith some embodiments. In the case of picking objects from a tote thenplacing it into another tote, intelligent perception and softwarealgorithms decide the best placement that saves space and maximizes thenumber of objects that can fit in the second tote. In the case ofpicking from a tote then placing it onto a shelf, software determinesthe geometry and pose of the object in hand such that the robotic systemreorients the object to put it inside the shelf unit without collision.Such a system saves space and ensures sufficiently large shelf unitswhere each represents a customer order.

In some embodiments, a robotic system comprising a robotic arm, anacquisition device (e.g., an end effector), an image sensor, and aprocessor determines a planned placement and a planned orientation of afirst object from a tote to a receptacle. The robotic system could alsobe system 100 described above. FIG. 4A depicts a robotic arm movingobjects from one tote to another tote. The planned placement is thetarget resting location of the first object in the receptacle, and theplanned orientation is the target resting orientation of the firstobject in the receptacle. First, the robotic system captures image datafor the first object using the image sensor. The image data may includemultiple images of the first object taken from multiple angles. Theimage data may be combined to form a three-dimensional map of the scene.Second, the robotic system determines a planned placement and a plannedorientation of the object relative to the receptacle using the imagedata such that a characteristic of the receptacle and its contents ismaximized.

In some embodiments, the receptacle is another tote, as depicted in FIG.4A. In such embodiments, the robotic system may aim to save space in theother tote. For example, it may place objects flush against the wall ofthe tote or against other objects already in the tote. The roboticsystem may aim to tightly pack objects to maximize space utilization sothat the tote or other container (or a box, gaylord, tote, bin, etc.)can hold a greater number of objects. There may be circumstances wherethis is different from saving space. The robotic system may optionallyemploy techniques used to solve the “knapsack problem” to optimize theuse of space in the other tote.

In some embodiments, the robotic system plans for several objects inadvance of placing any object rather than planning and placing oneobject at a time. The robotic system images several objects anddetermines their optimal placement and orientation before any objectsare placed. The system may employ dynamic programming or othertechniques to produce a plan. In such embodiments, the resultingarrangement of objects in the receptacle has the potential to save morespace and/or pack more objects in the receptacle compared to imaging andplanning one object at a time.

In some embodiments, the receptacle is a shelf, as shown in FIGS. 4B and4C. This may pose different constraints compared to a receptacle that isanother tote. For example, it is more difficult to stack objects on topof one another on a shelf without risking objects falling off. In suchembodiments, the system may exercise collision avoidance, where thecharacteristic of the receptacle and its contents is whether the objectwill collide with the receptacle or its current contents as the objectis placed in accordance with a given planned placement and a givenplanned orientation (with the goal being to ensure no collision takesplace). A collision may cause, for example, a stack of picked objects tofall down. In some embodiments, the system takes into account thephysical dimensions of the object. In other embodiments, the systemtakes into account other attributes of the object, such as its weight,center of gravity, and flexibility. The system may also determine aparticular planned movement of the robotic arm that avoid collision, ifseveral particular planned movement paths lead to the same plannedplacement and planned orientation but some paths might have collisionalong the way. In some embodiments, paths are planned to minimize totalpicking time. The shelf may optionally be configured to increase thestability of objects collected on the shelf. For example, as shown inFIG. 4C, the shelf is tilted back against the back wall.

FIGS. 4D-4K show snapshots of an exemplary picking of an object 170Afrom a tote 152A and the placement of a first object 170A onto a shelf,bin 182A. In FIG. 4D, tote 152A contains several objects 170 to bepicked and placed onto the shelf, bin 182A, which is tilting away fromthe robotic system 100. A robotic arm 102 with end effector 114 hoversabove tote 152A as the robotic system captures image data for firstobject 170A using an image sensor. In FIG. 4E, robotic arm 102 lowersinto tote 152A to acquire first object 170A using a suction. In FIG. 4F,robotic arm 102 picks up first object 170A and moves it toward a barcode scanner to be scanned. The movement includes translation, rotation,or a combination of both. In FIG. 4G, robotic arm 102 captures an imageof first object 170A in order to identify its barcode. In FIG. 4H,robotic arm 102 rotates and/or translates first object 170A andcontinues to capture multiple images of it as the system identifies andreads the barcode. The barcode indicates the action to be taken on firstobject 170A. In this case, first object 170A is to be placed on the toprightmost cubby on the shelf, bin 182A. In FIG. 4I, the systemdetermines a planned placement and a planned orientation of first object170A relative to bin 182A and robotic arm 102 begins to move firstobject 170A towards bin 182A in accordance with that planned placementand planned movement. In FIG. 4J, robotic arm 102 reaches bin 182A andreleases first object 170A into it. In FIG. 4K, robotic arm 102 returnsto tote 152A and is ready to begin picking and placing a second objectfrom the tote.

FIGS. 5A-5C are flow diagrams illustrating a method 500 of placingobjects in accordance with some embodiments. The method 500 isoptionally performed at a robotic system as described above withreference to FIG. 1 and FIGS. 4A-4K. Some operations in method 500 are,optionally, combined and/or the order of some operations is, optionally,changed.

FIG. 6A depicts an example of end effector 114 (see FIG. 1). The endeffector compliance is designed to minimize the risk of damage to theobjects (e.g., due to imperfect perception from sensor data, such ascaptured image data, and processing algorithms) while the robotic systemoperates at a high motion speed (e.g., the end effector can contact anobject at a higher speed when attempting to grasp the object). Inaddition to this feature or alternatively, the tote is attached to acompliant receptacle that effects the same relative motion between theend effector and the objects in the tote. In addition to avoiding thedamage, the compliance here also ensures stable and firm contact withthe object to guarantee a good seal or grip (in the case of gripper)despite the uncertainty of the exact location of the object due toimperfect image data and vision algorithms.

In this example of FIG. 6A, end effector 114 includes a suction gripper602. This gripper uses a tube 604 to produce suction at end 606 to gripobjects for picking and placing. End effector 114 also includes acompliance mechanism that allows for a greater tolerance in the amountof force that end effector 114 can apply to an object when attempting togrip the object. The compliance mechanism recoils when more than athreshold amount of pressure is applied to an object preventing damageto the object. For instance, when the contact force between an objectand the end of the end effector is greater than the gravitational forceon the moveable part of the end effector, the moveable part of the endeffector will recoil. The threshold amount of force can be adjusted byadding components (e.g., springs, elastic components, actuators) to theend effector, as described below.

In the example of FIG. 6A, the compliance mechanism includes track 608and optionally spring 610. As gripper 602 contacts an object, tube 604will slide up along track 608. Spring 610 allows for tuning of theamount of force applied before the compliance mechanism is activated. Asdepicted in FIG. 6A, spring 610 increases the force needed to activatethe compliance mechanism or alternatively provides additional force toreturn the moveable portion of the end effector after the force isremoved. By changing the spring connection, however, the force needed toactivate the compliance mechanism can also be reduced. Additionally, theamount of recoil can be tuned based on the movement tolerances of therobotic arm and errors associated with the vision system. In some otherexamples, the container for the objects being sorted (e.g., a tote) canalso be supported by a compliance member (e.g., springs, foam,hydraulics) on sorting stand 150 or as an integrated part of thecontainer. This compliance mechanism can be in place or in addition tothe compliance mechanism on end effector 114. Optionally, gripper 602can include a flexible suction nozzle on end 606 (see FIG. 1B). In somecases, end effector 114 allows for 100 mm or more of compliance.

FIGS. 6B-G depicts another example of end effector 620, with acompliance mechanism designed to minimize the risk of damage to anobject (e.g., a package to be picked up) while the robotic systemoperates at a high motion speed and to ensure stable and firm contactwith the object.

With reference to FIGS. 6B and 6C, the end effector 620 includes a rod624 and a casing 626. The end effector 620 can further include any typeof gripper (not depicted) attached to the distal end of the rod 624 forgripping and picking up objects. The end effector 620 can be attached toa motion device 630 (e.g., a robotic arm or a robotic arm phalange),such that the movement of the motion device causes movement of the endeffector 620, as discussed further in FIGS. 19A-19D. In the depictedexample in FIG. 6B, the robotic arm 630 moves downward, thus causing theend effector 620 to move downward until the distal end of the rod 624 ofthe end effector 620 comes in contact with a package 622.

If the robotic arm 630 continues moving downward when the distal end ofthe rod 624 is already in contact with the package 622, a compliancemechanism can be activated to prevent the distal end of the rod frommoving further to minimize damage to the package 622. With reference toFIG. 6C, the robotic arm 630 continues moving downward, causing thecasing 626 of the end effector attached to the robotic arm to movedownward by a distance D. However, the compliance mechanism has beenactivated such that the distal end of the rod 624 does not move further.As shown in FIGS. 6B and 6C, the distal end of the rod 624 remainsstationary, rather than colliding into the package 6C or pushing thepackage 622 away. Thus, the compliance mechanism minimizes damage to thepackage 622 even though the robotic arm 630 may operate (e.g., movingdownward) at a high motion speed.

FIGS. 6D and 6E illustrate an exemplary operations of the compliancemechanism. With reference to FIG. 6D, the end effector comprises asheath structure 628, which encloses a portion or a length of the rod622. The sheath structure comprises a slot 630. Further, the rod 624comprises a protruded piece 636 affixed to the rod, and the protrudedpiece 636 is positioned within the slot 630 of the sheath 628.

The sheath structure 636 is connected to the robotic arm 630 (FIG. 6B)and moves with the robotic arm. In other words, the sheath structure 636remains stationary relative to the robotic arm. In FIG. 6D, the roboticarm has moved such that the distal end of the end effector is in contactwith the object 622, similar to what is depicted in FIG. 6B.

When the robotic arm move further toward the package 622 and the distalend of the end effector is already in contact with the package 622, thecompliance mechanism is activated such that the end effector does notdamage the package 622. With reference to FIG. 6E, the robotic arm movesdownward toward to the package 622 and causes the sheath structure 628to move downward along the rod 624. The slot 630 of the sheath slidesalong the protruded piece 636 when the sheath structure 628 moves downthe rod. The distal end of the rod 624 remains stationary, rather thancolliding into the package 6C or pushing the package 622 away, similarto what is depicted in FIG. 6C.

As such, when the distal end of the end effector 620 contact the package622 at a higher speed when attempting to grasp the package, the endeffector does not collide into the package or push onto the package withsignificant force, thus minimizing the damage to the package. Thecompliance mechanism also ensures stable and firm contact with theobject 622 to guarantee a good seal or grip.

In some embodiments, the sheath structure is configured to start movingalong the rod when a resistance force between the distal end of the endeffector and the object is above a predefined threshold. In someembodiments, the predefined threshold is 0.

In some embodiments, the protruded piece is of a round shape. In someembodiments, the protruded piece is of a polygon shape. In someembodiments, the protruded piece 636 is affixed to a ring that wrapsaround the rod 624. As shown in FIGS. 6F and 6G, the protruded piece 636is affixed to a ring 634, which wraps around the rod 624. As the dottedarrows in FIG. 6F indicate, the sheath structure 628 is to be enclosedwithin the casing 626.

In some embodiments, the end effector is coupled to the motion devicevia one or more magnetic components on the end effector. As shown inFIG. 6F, the end effector comprises magnets 640 a, 640 b, and 640 c, forcoupling with the robotic arm. The magnetic coupling mechanism isdescribed in further detail below.

With reference to FIGS. 6D and 6E, in some embodiments, the end effectorcomprises a first end stopper 638 a and a second end stopper 638 b onthe rod. The two end stoppers are affixed to the rod. Further, thecasing 626 enclosing the sheath structure is placed between the firstend stopper and the second end stopper. As such, when the end effectoris not coupled to a motion device (e.g., a robotic arm), the endstoppers prevent the sheath structure and the casing from sliding offthe rod 624. When the end effector is coupled with the robotic arm(e.g., via magnets 640 a-c), the end piece 629 may be flush against thefirst end stopper 638 (as shown in FIG. 6D). When the compliancemechanism is activated, the casing 626 can move along the rod until thedistal end of the casing reaches the second end stopper 638 b or untilthe protruded piece 636 reaches the end of the slot 630.

In some embodiments, the distal end of the end effector comprises agripper. In some embodiments, the gripper comprises a suction cup, andthe hollow center of the rod 624 is configured to enable vacuumpass-through to produce suction at the distal end to grip objects forpicking and placing. In some embodiments, the distal end of the endeffector comprises other types of tools such as polishing tools, weldingguns, or a combination thereof.

In some embodiments, the system comprises one or more sensors fordetecting whether the compliant mechanism has been activated and howmuch retraction has occurred. The one or more sensors can includehall-effect sensors or inductance sensor. In some embodiments, the oneor more sensors are mounted on the sheath structure to detect relativemovement between the sheath structure and the protruded piece. Inaccordance with a determination that the compliant mechanism has beenactivated and/or the extent of retraction, the system can stop movingthe robotic arm to prevent a collision.

In some embodiments, when the object is picked up (e.g., via suctioncup) and lifted up in the air, the rod 624 can slide downward due togravity until the end piece 629 is in contact with the end stopper 638a, thus returning to the extended length shown in FIG. 6D.

FIGS. 7-9 illustrate exemplary systems for switching configurations ofan end effector between different types of grippers, such as fingergrippers and various suction nozzles, as well as between high-vacuum andhigh-flow suction systems in accordance with some embodiments. Therobotic system, such as system 100 described above, determines whethergripping at a particular location by a finger gripper or by a suctiongripper would be more effective, and if by suction, what suction nozzlesize is suitable. The robotic system contains both high-vacuum andhigh-flow suction systems. The system determines which of the twosystems to use given the task at hand, and switches between the twoaccordingly.

The end effector can switch configurations by changing the properties ofhow the end effector grips or grasps an object. For example, oneconfiguration of the end effector causes finger grippers to be usedwhile another configuration causes the end effector to use a suctiongripper. In this example, both grippers are coupled to the end effectorbut only one is configured for use at a time. Alternatively, only one ofthe grippers is present on the end effector and the two differentgrippers are automatically switched as necessary. In another example,two configurations for the end effector both use a suction gripper butthe suction gripper is configured differently in each case. Forinstance, one configuration uses a high-level vacuum for suction andanother uses a high-flow suction. These different configurations can beimplemented using two different suction sources (e.g., a vacuum sourceand a high-flow generator) and switched using a valve or othertechnique.

Embodiments of the robotic system determine probability maps for aplurality of different end effector configurations. The probability mapsdescribe the change of a successful grasp at various locations in ascene containing one or more (generally a plurality) of objects. Thesystem then picks a configuration based on any combination of factors,such as the configuration with the highest success probability, theconfiguration that will produce the fastest cycle time, the cheapestconfiguration, the configuration least likely to result in damage to anobject, or other similar factors.

FIG. 7 illustrates a robotic system according to various embodiments ofthe present disclosure. Robotic system 800 (partially shown) includes anend effector 802 having two fingers, a first finger 806 and a secondfinger 808, where the robotic system 800 is configured to pick an objectby gripping the object 809 using the two gripper fingers 806 and 808 ofthe gripper end effector 802. Robotic system 800 also includes a suctiongripper end effector 804 having a suction nozzle 805 and one or moresuction generators (not shown), where the robotic system 800 isconfigured to pick an object using the suction gripper having a suctionnozzle 805. In some embodiments, the suction nozzle 805 can have varioussizes and shapes.

FIG. 8 illustrates exemplary motion primitives of the robotic systemaccording to various embodiments of the present disclosure. Grip downmotion primitive 812 optionally grips object 810 vertically using thegripper end effector 802 having two parallel fingers 806 and 808. Thegrip down motion primitive 812 can be used to pick up objects withsmaller, irregular surfaces (e.g. small tools, deformable objects), ormade of semi-porous materials that prevent a suction seal (e.g. cloth).Flush grip motion primitive 814 is optionally similar to the grip downmotion primitive 812, but with the additional motion of using a flexiblespatula 807 attached to the first finger 806 of the grip end effector802 to slide the finger 806 between the object (e.g., 810) and a side ofthe receptacle 819 containing the object 810. Suction down motionprimitive 816 optionally picks the object 810 vertically by placing thesuction end effector 804 substantially vertically on the object 810 andgenerating a suction force. The suction down motion primitive 816 iseffective for picking objects with large and flat suctionable surfaces(e.g. boxes, books, wrapped objects) or objects in heavy clutter (e.g.,among a large number of other objects in the scene). Suction side motionprimitive 818 optionally picks the object from an angle that is notsubstantially vertical by approaching with a suction end effector tiltedat an angle other than substantially vertical. The suction down motionprimitive is effective for picking thin and flat objects resting againsta side of a receptacle containing the object, which may not havesuctionable surfaces from the top.

FIG. 9 illustrates exemplary processes by which robotic system 100determines to pick an object using a motion primitive associated with asuction gripper end effector (e.g., 804) according to variousembodiments in the present disclosure. A simulation 821 of a roboticsystem 100 picking scene 822, i.e., a representation (e.g., visualrepresentation, such as an image) is shown. Scene 822 of the physicalenvironment in which the robotic system operates) optionally includes aplurality of objects (e.g., a cluttered scene). The robotic system(e.g., 100) optionally determines, using machine learning algorithms824, a plurality of pixel-wise probability density maps 826 of scene822, where each probability density map corresponds (one-to-one in someembodiments) to a motion primitive associated with the suction endeffector (e.g., suction down primitive 816 or suction side primitive818). A probability density map 826 may optionally be depicted as agrayscale image where, for example, regions 827 a with 0% probability ofsuccess are labeled black, regions 827 b with 100% probability ofsuccess are labeled white, and regions 827 c with probabilities ofsuccess between 0% and 100% are in shade of grays that correspond tothose probabilities. These pluralities of pixel-wise probability mapsare translated into a plurality of maps that categorize each pixel. Map828, which corresponds to scene 822 and is generated from the pluralityof probability density maps 826, shows regions 829 a (e.g. representedas blue) that are categorized as candidate locations to attempt picking,and subsets 829 b of those regions (e.g. represented as magenta) thatare categorized as the best locations to attempt first. In someembodiments, the robotic system 100 then chooses a suction motionprimitive (e.g., suction down primitive 816) and a proposed suctionpoint location to execute the chosen motion primitive. In otherembodiments, the robotic system 100 executes one or more of the pushing,toppling, and pulling primitives on one or more objects to rearrange thescene before executing a suction motion primitive, which makes one ormore better grasp points accessible to the robotic arm.

In some embodiments, robotic system 100 is configured to determinepicking an object using a motion primitive associated with a grip endeffector (e.g., 802) according to various embodiments in the presentdisclosure. In some embodiments the robotic system (e.g., 800)optionally processes a representation of the scene 822 by rotating therepresentation around the vertical axis at various angles, anddetermines a pixel-wise probability map of the scene for each gripperorientation. The robotic system 100 optionally determines a verticalcolumn of 3D space in a 3D representation of scene as a 3D griplocation. In some embodiments the robotic system 100 optionallydetermines, using machine learning algorithms, a plurality of pixel-wiseprobability maps of the scene 822. In some embodiments the roboticsystem 100 then chooses a grip motion primitive (e.g., grip downprimitive 812) and a proposed grip point location to execute the motionprimitive.

In various embodiments of the present disclosure, a robotic system canuse suction or grip to pick an object. In various embodiments, therobotic system can further determine whether to use suction or grip topick an object, what motion primitive to use, among other parameters. Insome embodiments, the robotic system optionally has one or more arms. Insome embodiments the robotic system can operate in one or more axes(i.e., degrees of freedom). In some embodiments, the robotic systemoptionally operates with six axes (i.e., a six-axis robotic system). Insome embodiments, a robotic system (e.g., a pick and place roboticsystem 800) optionally has a first end effector (e.g., 804) and a secondend effector (e.g., 802). In some embodiments, the robotic system (e.g.,800) optionally includes one or more end effectors configured tomanipulate (e.g., pick, move, rotate, or place) an object. In someembodiments, when performing a task (e.g., picking an object having acertain size, shape, or composition, etc.) the robotic system (e.g.,800) optionally chooses an end effector among multiple end effectors.

In some embodiments, the first end effector (e.g., 804) is optionallyconfigured to pick an object using suction, and the second end effector(e.g., 802) is configured to pick the object using grip (that is, eachend effector is configured to pick an object using either suction (e.g.,suction end effector) or grip (e.g., grip end effector). In someembodiments a suction end effector optionally picks up an object bypressing against a surface of the object and creating a suction forcethrough the end effector. In some embodiments, a grip end effector usesa plurality of (e.g., two) grippers to grip the object.

In some embodiments, the robotic system (e.g., 800) optionallydetermines a plurality of probability maps of a scene including theobject and at least one other object. In some embodiments, the scene(e.g., 822) includes the object to be picked and at least another object(e.g., the scene is cluttered). In some embodiments, the object and theanother object can be different types. In some embodiments, eachprobability map corresponds respectively to a different motion primitiveamong a plurality of motion primitives. In some embodiments, the roboticsystem (e.g., 100 or 800) has one or more motion primitives (e.g.,predefined motions that the robotic system can take). In someembodiments, each motion primitive is associated with a correspondingprobability map (e.g., 826 or 836). In some embodiments, each motionprimitive and its corresponding probability map optionally has aone-to-one correspondence.

In some embodiments, each motion primitive is associated with using thefirst end effector (e.g., 804) or the second end effector (e.g., 802) topick the object; in other words, each motion primitive of the roboticsystem is associated with picking the object using either a suction endeffector (e.g., a suction motion primitive such as 816 or 818) or a gripend effector (e.g., a grip motion primitive such as 812 or 814).

In some embodiments, the robotic system (e.g., 800) optionally chooses amotion primitive among a plurality of motion primitives to use inpicking the object (e.g., 810) based on the plurality of probabilitymaps. In some embodiments, the robotic system (e.g., 800) optionallychooses the motion primitive whose corresponding probability mapindicates the highest likelihood of picking the object (e.g., 810)successfully.

In some embodiments, the plurality of motion primitives optionallyinclude a first motion primitive (e.g., 812) using the second endeffector (e.g., 802). In some embodiments, the plurality of motionprimitives optionally include a second motion primitive (e.g., 814)different from the first motion primitive (e.g., 812) using the secondend effector. In some embodiments, the first motion primitive (e.g.,812) and the second motion primitive (e.g., 814), different from thefirst motion primitive (e.g., 812), are optionally associated with usingthe suction end effector to pick the object. In some embodiments, theplurality of motion primitives optionally includes a third motionprimitive (e.g., 816) using the first end effector (e.g. 804) and afourth motion primitive (e.g., 818) different from the third motionprimitive using the first end effector. In some embodiments, the thirdmotion primitive and the fourth motion primitive, different from thethird motion primitive, are optionally associated with using the gripend effector to pick the object.

In some embodiments, the plurality of motion primitives optionallyincludes a gripping down motion primitive using the second end effector(e.g., 802), which grips objects vertically using a grip end effector(e.g., 802) having two parallel gripper fingers (e.g., 806 and 808). Insome embodiments, the plurality of motion primitives optionally includesa flush gripping motion primitive (e.g., 814) using the second endeffector (e.g., 802). In some embodiments, the flush gripping motionprimitive (e.g., 814) is similar to the grip down motion primitive, butwith the additional motion of using a flexible spatula (e.g., 807)attached to the grip end effector to slide one gripper between theobject (e.g., 810) and a side of a receptacle (e.g., 809) containing theobject. In some embodiments, the plurality of motion primitivesoptionally includes a suction down motion primitive (e.g., 816) usingthe first end effector (e.g., 804). In some embodiments, the suctiondown motion primitive optionally picks the object (e.g., 810) verticallyby placing a suction end effector substantially vertically on the object(e.g., 810) and generating a suction. In some embodiments, the pluralityof motion primitives optionally includes a suction side motion primitive(e.g., 818) using the first end effector (e.g., 804). In someembodiments, the fourth motion primitive associated with using thesuction end effector to pick the object (e.g., 810) is optionally asuction side motion primitive. In some embodiments, the suction sidemotion primitive optionally picks the object (e.g., 810) from an anglethat is not substantially vertical by approaching with a suction endeffector tilted at an angle other than substantially vertical.

In some embodiments, the robotic system (e.g., 800) further determines aplurality of probability maps (e.g., 826 and 836) of the scene includingthe object (e.g., 810) and at least another object

In some embodiments, the robotic system (e.g., 800) has not determinedpreviously a probability map of a scene including the object (e.g.,810). In other words, the object (e.g., 810) to be picked and otherobjects in the scene are optionally novel, and the robotic system (e.g.,800) can pick the object (e.g., 810) without any a priori training forobjects that have not previously appeared in a scene for which therobotic system (e.g., 800) has determined a probability map.

In some embodiments, the plurality of probability maps are optionallypixel-wise probability maps. In other words, the robotic system (e.g.,800) optionally assigns a probability value to each pixel of a digitalimage of the scene. In some embodiments, the plurality of probabilitymaps are pixel-wise binary probability maps (i.e., positive ornegative).

In some embodiments, the robotic system (e.g., 800) optionallydetermines a probability map of the scene (e.g., 836) corresponding to amotion primitive associated with the first end effector (e.g., 804) bydetermining a proposed suction point (e.g., a three-dimensional positionwhere a suction end effector should come in contact with the object'ssurface in order to pick and lift it successfully) corresponding to apixel of an image of the scene (i.e., each pixel corresponds to adifferent position on which to execute the primitive), a localgeometrical quantity of the proposed suction point (e.g., computed froma projected 3D point cloud), and a probability (e.g., between zero andone, where a value closer to one implies a more preferable suctionpoint) of picking the object (e.g., 810) at the proposed suction pointusing a machine learning algorithm (e.g., 826). In some embodiments, therobotic system (e.g., 800) optionally outputs a pixel-wise binaryprobability map of the scene. In some embodiments, the pixel-wise binaryprobability map of the scene is optionally based on the determination ofthe proposed suction point, the local geometrical quantity of theproposed suction point, and the probability of pick the object (e.g.,810) at the proposed suction point.

In some embodiments, the first end effector (e.g., 804) is optionallyconfigured to couple with a first attachment (e.g., suction nozzle 807).In some embodiments the robotic system (e.g., 800) optionally chooses asuction nozzle among multiple available suction nozzles having varioussizes, shapes, etc., and optionally couples the suction nozzle with thesuction end effector. In some embodiments, the robotic system (e.g.,800) optionally determines a probability map of the scene thatcorresponds to each configuration of a suction nozzle coupled with thesuction end effector. In some embodiments, the robotic system (e.g.,800) optionally determines a plurality of probability maps of a scene bydetermining a first probability map of the scene corresponding to amotion primitive associated with the first end effector (e.g., 804)coupled with the first attachment (e.g., suction nozzle 807).

In some embodiments, the robotic system (e.g., 800) optionally includesa first suction generator (e.g., a vacuum pump) configured to generatesuction and a second suction generator different from the first suctiongenerator. In some embodiments, the first suction generator and thesecond suction generator are of different types (e.g., a vacuum pump anda high-flow air blower or exhauster). In some embodiments, the roboticsystem (e.g., 800) chooses a motion primitive among a plurality ofmotion primitives to use in picking the object (e.g., 810) based on theplurality of probability maps by: in accordance with the chosen motionprimitive (e.g., 816 or 818) being associated with using the first endeffector (e.g., 804), determining whether to generate suction using thefirst suction generator (e.g., a vacuum pump) or the second suctiongenerator (e.g., a high-flow air blower or exhauster). In other words,in some embodiments where the robotic system (e.g., 800) chooses amotion primitive associated with using the suction end effector to pickthe object (e.g., 810) (e.g., the suction down motion primitive 816 orthe suction side motion primitive 818), the robotic system (e.g., 800)optionally determines which type of suction generator should be used togenerate the suction force. In some embodiments, because different typesof suction generators have different suction force characteristics(e.g., a high-flow suction generator is optionally more effective than avacuum pump suction generator for picking an object with a poroussurface such as fabrics), the robotic system (e.g., 800) optionallydetermines a probability map of the scene that corresponds to eachconfiguration of the suction end effector coupled with the vacuum pumpsuction generator or the high-flow suction generator.

In some embodiments, the robotic system (e.g., 800) optionally includesa first sensor measuring a first property associated with the firstsuction generator, and a second sensor measuring a second propertydifferent from the first property associated with the second suctiongenerator. In some embodiments, where the robotic system (e.g., 800)uses a suction motion primitive (e.g., suction down primitive or suctionside primitive) to pick an object and where the robotic system (e.g.,800) optionally includes both a high-level vacuum pump suction generatorand a high-flow air exhauster suction generator, different types ofsensors are optionally used to measure the suction when the suction endeffector is in contact with the object. In some embodiments, inaccordance with determining to generate suction using the first suctiongenerator, the robotic system (e.g., 800) optionally determines a firstend effector's suction grip based on the first property measured at thefirst sensor, and in accordance with determining to generate suctionusing the second suction generator, the robotic system (e.g., 800)optionally determines the first end effector's suction grip based on thesecond property measured at the second sensor. For example, in someembodiments, where a vacuum pump suction generator is used to generatesuction, a vacuum pressure sensor is optionally used to measure the airpressure; the air pressure is in turn indicative of the level of suctiongrip the suction end effector has on the object (e.g., 810) (e.g., lowerair pressure indicates a more secure suction grip, and vice versa). Insome embodiments, where a high-flow suction generator is used togenerate suction, a mass airflow sensor is optionally used to measurethe air flow rate, which is in turn indicative of the level of suctiongrip the suction end effector has on the object (e.g., 810) (e.g., lowerair flow rate indicates a more secure suction grip, and vice versa).

In some embodiments, the robotic system (e.g., 800) optionallydetermines a probability map of the scene corresponding to a motionprimitive associated with the second end effector (e.g., 802) bydetermining a proposed three-dimensional (3D) grip locationcorresponding to a three-dimensional representation of the scene. Insome embodiments, the robotic system (e.g., 800) optionally determines avertical column of 3D space in the scene as a 3D grip location. In someembodiments, the robotic system (e.g., 800) optionally determines amiddle point between a first gripper and a second gripper of the secondend effector. In some embodiments, the robotic system (e.g., 800)optionally determines an angle corresponding to the orientation of thefirst gripper and the second gripper (e.g., an angle which defines theorientation of the gripper around the vertical axis along the directionof gravity). In some embodiments, the robotic system (e.g., 800)optionally processes the visual representation of the scenes to accountfor a plurality of different gripper orientations around the verticalaxis and determines a pixel-wise probability map of the scene for eachgripper orientation. In some embodiments, the robotic system (e.g., 800)optionally determines a width between the first gripper and the secondgripper at the proposed grip location (e.g., based on the 3D visualrepresentation of the scene). In some embodiments, the robotic system(e.g., 800) optionally determines a probability of picking the object(e.g., 810) at the proposed three-dimensional location. In someembodiments, the robotic system (e.g., 800) optionally outputs apixel-wise binary probability map of the scene. In some embodiments, therobotic system (e.g., 800) outputs a pixel-wise binary probability mapof the scene based on the determination of the proposed grip point, thelocal surface geometry of the proposed suction point, and theprobability of picking the object (e.g., 810) at the proposed suctionpoint.

In some embodiments, the robotic system (e.g., 800) optionallydetermines the distance between the proposed three-dimensional griplocation relative to a side of a receptacle containing the object (e.g.,810) and the at least one other object. That is, the robotic system(e.g., 800) optionally determines the distance of each proposed grippoint from sides of the receptacle. In some embodiments, the roboticsystem (e.g., 800) optionally determines whether to use the third motionprimitive (e.g., 812) or the fourth motion primitive (e.g., 814) basedon the plurality of probability maps, the width between the firstgripper and the second gripper at the proposed grip location, and thedistance between the proposed three-dimensional grip location relativeto a side of a receptacle containing the object (e.g., 810) and the atleast one other object. In some embodiments, the robotic system (e.g.,800) optionally determines whether to use the grip down motion primitiveor the grip side motion primitive based on the probability maps, and thedistance between the proposed three-dimensional grip location relativeto the sides of the receptacle containing the object (e.g., 810) andother objects.

FIGS. 10A-10C illustrate exemplary systems for detecting barcodes inorder to avoid picking on them in accordance with some embodiments. Therobotic system takes an image and then identifies a location forpicking, either by suction or with a gripper as the end effector. Thelocation is selected based on a success probability map across thepixels of the image, wherein the probability map is generated from amachine learning algorithm.

In some embodiments, the robotic system comprises a robotic arm, anacquisition device (e.g., an end effector), an image sensor, a database,and a processor. The robotic system determines a location in a scene,wherein the scene includes a plurality of objects to be picked, for arobotic system to acquire an object in the plurality of objects. Theremay be no specific instruction as to which object is picked up first.Such a system may have been trained on a dataset where users haveindicated suitable locations to pick up objects in various scenes. Thisdataset is stored in the database.

First, the system captures image data for the scene using the imagesensor. The image sensor may be a depth-sensing device that producesdepth information for image recognition and other artificialintelligence techniques in addition to taking images. Second, the systemgenerates, based on the image data for the scene, a probability mapcomprising a plurality of probabilities each corresponding to a regionin a plurality of regions on the object, wherein the plurality ofprobabilities is based on the likelihood that the corresponding regionin the plurality of regions is a barcode portion, and data stored in thedatabase. The resolution of the probability map may be as high as theimage data. In other embodiments, the resolution of the probability mapis on par with the resolution of the robotic arm and acquisition device(e.g., an end effector). In some embodiments, the probabilities aredetermined with machine learning algorithms trained on the datasetstored in the database.

Next, the system determines a location on the object by selecting aregion on the object corresponding to a probability in the plurality ofprobabilities that exceeds a threshold probability. The thresholdprobability indicates an acceptable chance that attempting to pick upthe object at that location will succeed. The threshold probability maybe, for example, 90%, 95%, or 99%. In some embodiments, the user may setthe threshold probability. In other embodiments, the thresholdprobability is determined by other constraints on the system, such asmaximum permitted time to successfully pick up an object. In otherembodiments, the threshold probability adjusts dynamically during thepicking process.

In some embodiments, in accordance with a determination that theacquisition device coming into contact with the object at a regioncauses the barcode on the object to be occluded, a probability of zerois assigned to the region. In this sense, a barcode is “occluded” whenone or more bars in the barcode are completely covered. Note thatpartial coverage of one of more bars is acceptable if at least a portionof each bar is readable. In such embodiments, the probability that thebarcode cannot be read is minimized. For example, FIG. 10B depicts abarcode 1020 that is partially covered by acquisition device contactarea 1022. However, since all the bars have portions that are notcovered, the entire barcode is still readable when scanned across itstop row, by, for example, the Cognex device 1010 depicted in FIG. 10Awith scanner 1012. In contrast, FIG. 10C depicts a barcode 1024 that isoccluded: Even though some bars are unaffected by acquisition devicecontact area 1026, there are some bars that are completely covered byacquisition device contact area 1026, so the barcode cannot beinterpreted when scanned in any direction.

In some embodiments, the system leverages dense probabilities for speedpicking by attempting multiple different acquisitions in quicksuccession until at least one of them is successful. The systemdetermines a plurality of locations on the object by selecting regionson the object corresponding to probabilities in the plurality ofprobabilities that exceeds a threshold probability, wherein the distancebetween the locations is beyond a threshold distance.

In some embodiments, the system enters a troubleshooting mode if itcannot find the barcode on an object or if none of the probabilities areabove a threshold. The system may not be able to find the barcode if,for example, the surrounding lighting and shadows from other objectsmake the barcode difficult to detect. The barcode may even be missing.In some embodiments, a system in troubleshooting mode will make anotherattempt at finding the barcode by picking up the object, rotating it,and putting it back into the scene. The system will then repeat theprocess of capturing image data for the scene and generating aprobability map. In other embodiments, a system in troubleshooting modeavoids repeating unsuccessful attempts. For example, after anunsuccessful attempt to pick up the object, the system sets theprobabilities corresponding to regions within a threshold radius of theattempted location to zero. As discussed above, locations close to oneanother tend to have highly correlated probabilities of success. Thus,moving to a different region further away from a failed location is morelikely to result in a successful pick up location.

FIGS. 11A-11C are flow diagrams illustrating a method 1100 of pickingbased on barcode scanning in accordance with some embodiments. Themethod 1100 is optionally performed at a robotic system as describedabove with reference to FIG. 1 and FIGS. 10A-10C. Some operations inmethod 1100 are, optionally, combined and/or the order of someoperations is, optionally, changed.

FIG. 12 illustrates exemplary systems for barcode scanning in accordancewith some embodiments. The robotic system identifies the optimal way tomove the object to increase the likelihood that the bar code can be seenin a scan station. The total time it takes the system to rotate theobject, image it, and scan the bar code is minimized.

In some embodiments, a robotic system comprising a robotic arm, agripper, and an image sensor scans a barcode on an object. First, thesystem grips the object using the gripper. This may be accomplished, forexample, by the means described earlier, including methods thatdetermine which location to acquire an object and whether to use agripper. Next, the system estimates the location of a barcode on theobject. This estimation may include, for example, previous images thesystem has of the scene. Alternatively, the system may capture images orvideo as the object is gripped and estimate the location of the barcodefrom the images or video. Since these initial images and videos areintended to merely estimate the location of the barcode and not to readthe barcode, the system may use lower resolution image sensors for thistask in order to save energy, processing power, and/or time. Forexample, a low-resolution camera may capture an image of the scene froma top view. If no barcode is detected on the top face of an object, thesystem may estimate that the barcode is on one of the other faces of theobject. In some embodiments, the system assigns probabilities tolocations on the object where the barcode may lie.

The system then determines a planned movement of the object, wherein theplanned movement comprises translation and rotation; and the plannedmovement is based on the location of the image sensor relative to theestimated location of the barcode on the object. For example, if thebarcode is estimated to be at the bottom of the object and the imagesensor is on the side, then the planned movement is to rotate the object90 degrees from the bottom to the side.

The system then moves the object in accordance with the plannedmovement. Due to the planned movement, the barcode is now more likely tobe in view of the image sensor than it was before the planned movement.The system then captures image data for the object using the imagesensor. Next, the system identifies a barcode on the object using theimage data. This may include, for example, applying image recognitiontechniques to the image data. Finally, the identified barcode on theobject is scanned. In some embodiments, the image sensor and the barcodescanner are the same device.

In some embodiments, the robotic system further comprises a plurality ofbar code scanners, each aligned at different angles and orientations.Such an arrangement enables the system to collectively capture most orall possible locations of barcodes. The system may be configured toexecute all the scanners simultaneously in order to save time. In theevent that more than one scanner reads a barcode, the multiple readingsmay act as an error check. For example, if two scanners agree on abarcode reading but a third scanner does not, the system may use thereading agreed upon by the first two scanners and ignore the readingfrom the third scanner. In an alternative embodiment, if the barcodereadings are not unanimous across all scanners that have registered areading, then the system may re-scan the barcode or may ask for humanassistance to resolve the discrepancy. In some embodiments, the systemis configured to detect objects that have multiple different barcodesattached. Under such circumstances, the system may ask for humanassistance.

In some embodiments, the robotic system uses mirrors to increase theprobability that the image sensor can capture the barcode. Such a systemfurther comprises one or more mirrors configured so that an image of theobject and its surroundings includes the reflection of the one or moremirrors, wherein reflections include surfaces of the object not directlyvisible from the image sensor. Thus, if the estimated position of thebarcode is incorrect, there is an improved probability that the barcodewill be visible in one of the surfaces reflected by a mirror to theimage sensor. One such system is depicted in FIG. 12, where two mirrors1212 and 1214 are configured behind object 1210 at an angle so that theimage sensor 1010 (a bar code scanner, such as a Cognex scanner) is ableto capture multiple sides of the object in a single image capture. Insome embodiments, the mirrors are configured so that most or allpossible locations of barcodes appear in the resulting image. In suchembodiments, the mirrors are placed in a manner to optimize theprobability that the bar code is visible in a single image capture thatincludes the object and reflections of the object in the mirrors.Otherwise, if the first image shot fails to find the barcode in thefirst image, the system may move the object and capture a second image.

In some embodiments, the planned movement is set to minimize the time tocomplete the planned movement subject to a constraint. For example, if aplanned movement comprises both a non-zero rotation and a non-zerotranslation, it is efficient to execute both degrees of motionsimultaneously. However, if such an execution would cause the object totouch a second object, then a different execution is followed, e.g., onewhere the object moves around the second object. In some instances, themotion speed and/or perception algorithm is adjusted for delicate objecthandling (i.e., to provide for more conservative operation of thesystem). In some embodiments, the constraint is that the probability ofthe object falling off of the robotic arm is below a threshold. Thesystem accordingly adjusts the motion speed and perception algorithmsfor delicate object handling. In some embodiments, the minimization oftime to complete the planned movement subject to a constraint isdetermined by machine learning algorithms.

In some embodiments, when the system detects no valid barcode afterscanning, it enters a troubleshooting mode. In some embodiments, thetroubleshooting mode entails trying again: the system rotates theobject, puts the object back into the scene (presumably at a differentorientation from where it was originally before it was picked up thefirst time), and grips the object again. Additionally or alternatively,the system may perform non-gripping actions such as toppling, pulling,and/or pushing to rearrange the objects in the bin to increase thechance of a successful grasp. It is possible that the barcode was in aninaccessible orientation the first time, and a second barcode scan maybe successful. However, in some embodiments, after a threshold number offailed attempts, the system may enter a second troubleshooting mode,which comprises alerting a user for assistance.

In some embodiments, the robotic system has external lights that areintelligently controlled to improve the visibility of the barcode on anobject. For example, if the robotic system is in a dimly lit location orif obstacles nearby otherwise cast shadows into the barcode scanningarea, the robotic system may increase the intensity of the externallights. Conversely, if the surface of the object on which the barcodesits is somewhat reflective, the robotic system may decrease theintensity of the external lights. In some embodiments, the externallights are each configured at different angles to focus on differentsurfaces of the object and are independently controlled. In someembodiments, a plurality of light sensors detects the intensity of lightaround the barcode scanning region and feeds information to the roboticsystem to intelligently control the external lights.

FIGS. 13A-13D are flow diagrams illustrating a method 1300 of barcodescanning in accordance with some embodiments. The method 1300 isoptionally performed at a robotic system as described above withreference to FIG. 1 and FIG. 12. Some operations in method 1300 are,optionally, combined and/or the order of some operations is, optionally,changed.

FIGS. 14A-B depicts pick and place system 1400. As shown in FIG. 14A,system 1400 includes robotic arm 1402 (which is, for example, the sameor similar to robotic arm 102 described above with respect to FIG. 1 oranother motion device); tote conveyor 1404 (which, for example,transports receptacles (e.g., totes) to a sorting position adjacent torobotic arm 1402; and put wall 1406 (which, for example, includes binsthat receive items that are sorted from the receptacles by the roboticarm and control system (not shown)). User 1408 supervises, operates,and/or otherwise interacts with pick and place system 1400 through anuser interface (not shown) or may manually interacting with the roboticarm, totes, put wall, and/or items that are going to be sorted, areactively being sorted, or have been sorted. FIG. 14B depicts barcodescanners 1420 and 1422 (e.g., similar as or the same as barcode scanner1012 (FIG. 10A)) and 3D camera 1424 (e.g., similar as or the same ascamera 158) that is part of a vision system (e.g., similar as or thesame as the vision system described with respect to FIG. 1).

Referring again to FIG. 14A, tote conveyor 1404 includes input conveyor1410 that directs totes, such as input tote 1412, to a location in frontof robotic arm 1402. Justifying conveyor 1414 aligns totes to roboticarm 1402 and/or put wall 1406. In some cases, justifying conveyor 1414aligns totes under optional funnel 1416 that helps redirect items thatare dropped, either intentionally or inadvertently, by robot arm 1402.Output conveyor 1418 directs totes, e.g., after its contents are donebeing sorted, away from robotic arm 1402 so that another tote and bepositioned so that its contents may be sorted. Input conveyor 1410,justifying conveyor 1414, and/or output conveyor 1418. Each conveyor canbe implemented with any number of components that move and/or positiontotes. In one case, the conveyors are implemented with passive rollersto allow totes to move in response to gravity or external sources (e.g.,user 1408 or a robot pushing the tote). In another case, the conveyorsare implemented with driven rollers or other active components that moveand/or position totes along the conveyors. The conveyors can alsoinclude belts, bearings, chutes, tracks, and/or any combinations theseand/or other components.

In some cases, tote conveyor 1404 is modular and that one or morecomponents of tote conveyor 1404, such as input, justifying, and/oroutput conveyors are modules that are positioned together (and in somecases coupled together or with other components in the system).Optionally, tote conveyor 1404 automatically feeds the tote (followingthe arrow in FIG. 14A) to the work area in front of robotic arm 1402. Inthe previous design, it was done by a person. The automated system caninclude tamper, pop-up cates, photo sensors, and conveyor rollers thatare adjustable to allow precise justification of the tote.

Totes are fed into the system by various means, including anotherconveyor carrying totes containing items from a batch-picked(wave-picked) order that are transitioned onto input conveyor 1410 oronto justifying conveyor 1414. Human operators can also provide totes tothe system by carrying (e.g. with a cart) totes onto input conveyor 1410or onto justifying conveyor 1414. An autonomous ground vehicle couldalso be used to supply totes in a similar manner (e.g., loading themonto input conveyor 1410 or onto justifying conveyor 1414).

FIG. 15 depicts a side view of put wall 1406 that includes chute 1500that directs picked items on to moving shelf 1502. Put wall 1406optionally includes optical sensors 1504 (e.g., photo diodes, imagesensors, or optical sensors) that detect when an item is successfullydirected onto moving shelf 1502. The sensor can be placed at theentrance of a bin or in the middle through holes or in between the shelfand the chutes.

FIG. 16 depicts justifying conveyor 1414 connected to put wall 1406. InFIG. 16, put wall 1406 include wheels 1600 that allow for a human orautonomous robot to move put wall 1406 to another location, such as apacking station) after all picked items have been placed on moving shelf1502. Alternatively, if moving shelf 1502 includes removable bins orother containers, a human operator or autonomous robot can move the binsor other containers to another conveyor or vehicle to be taken anotherlocation, such as a packing station. The bins on the moving shelf mayhave damping foam on the bottom, shocks, or other components that reduceimpacts of picked items that are dropped into a particular bin.

FIG. 16 also include funnel 1416 for fault recovery. It redirectsmisplaced items back into the tote to recover put wall 1406 placingactions that fail. Funnel 1416 is designed to avoid occlusion of machinevision cameras and prevent dropped items from landing on justifyingconveyor 1414. In some cases, funnel 1416 is lined with a soft and/orslippery material to facilitate return of items to the tote. In somecases, funnel 1416 has asymmetric sloped sides that provide a largerarea to catch items next to put wall 1406 than next to robotic arm 1402or adjacent sides.

FIG. 17 depicts a variation of put wall 1406 where chute 1500 isdetachable and justifying conveyor 1414 is separate from the rest of putwall 1406 (e.g., moving shelf 1502). Moving shelf 1502 can be detached,rolled away to a packaging station, and replaced with an empty movingshelf. For example, Moving shelf 1502 can be unlocked from the packingstation via a lock mechanism (e.g., latches, fasteners, or magnets). Insome cases, after robotic arm 1402 (FIGS. 14A-B) finishes a pickingtask, a human operator (e.g., user 1408) will move the shelf to anotherpackaging station. Alternatively, automated machinery move the shelf toanother location (e.g., by lifting it from the bottom and carrying itaway).

A vibrator may be attached or coupled to the tote to vibrate it so thatthe objects inside the tote are separated and/or spread out (e.g., amore flat arrangement). In some cases, this will reduce the difficultyfor a suction cup end effector to grip the objects.

Exemplary pick and place system 1440 sorts a tote (or other container)of objects that need to be sorted (batch-picked warehouse SKUs orparcels in a sortation center) comes from the system's upstream (e.g.,conveyor, human, or mobile robot). When the tote arrives, the systemadmits it or temporarily buffers it. The system performs a pick process(e.g., similar to the same as the pick processes described above).

After identifying a target pick location, the system picks up anassociated object and lifts it in the air while moving it during barcodescanning and pose estimation. The system will put the object back intothe tote (or another container to let a human or a machine to handle itlater) in the case of an unsuccessful barcode scanning, detection ofdouble picking, unsuitable pose, or other error so that another objectcan be picked for placing into the chute next. When putting the objectback to the tote, the system will calculate a preferred pose to place sothat it does not create a tall stack, which complicates the later picks.

When a successful pick and object identification (e.g., by a barcodescan) occurs, the system places the object into the correspondingdestination chute bin opening. For example, the destination is chosenbased on whether items associated with the picked item already reside orwill reside in the future or the destination is chosen based on a chutefor a bin that has been assigned an order to which the picked objectbelongs. To assist in placing the object, the system optionally usescameras to estimate pose, orientation, or other characteristics of thepicked object in order to successfully place the object in thedestination chute in opening (e.g., ensuring that the object will fit inthe chute opening and/or reducing the likelihood of the object collidingwith the chute opening). Additionally, in some cases the chute hasoptional optical sensors to confirm the object has fallen into thecorrect chute bin opening. The object falls into the bin on the mobileshelf aligned to the chute.

After finishing sorting, a human may carry the mobile shelf to thepackaging station or directly take things out of the bins of the bins onthe shelf. A mobile robot may also take carry the shelf away. Thehuman-machine interface (e.g., see FIG. 18) has a teleoperationinterface to guide the robot picking in the case of repeated failures.

FIG. 18 depicts user interface 1800 for a human operator to easilycontrol pick and place system 1400. In some cases, the focus ofinterface 1800 is facilitating system start, reset and fault recovery,e.g., with command buttons (e.g., physical or virtual) 1801-1803. Userinterface 1800 optionally also provides for real time video 1804 of thesystem operation and/or system messages 1805 to the human operation. Thesystem will also display alert information to the operator should thesystem detects error that needs human intervention or double checking.User interface also includes a teleoperation functionality to supportpicking in the case of repeated failures. In some instances, theinterface allows an operator to guide the picking of an object in thecase of repeated failures by the robot to automatically pick the object.

FIGS. 19A-D depict compliant magnetic coupling for robotic arm endeffectors. In FIG. 19A, end effector 1902 (which in some cases is thesame or similar to end effector 114 of FIG. 1 or other end effectorsdescribed above) includes a rod 1907 and a suction gripper 1904. The endeffector 1902 is attachable to the distal surface 1900 of robotic armphalange 1906 of robotic arm 102 via a compliant magnetic couplingmechanism.

The compliant magnetic coupling mechanism allows for breakaway whenexcessive force is applied to robotic arm 102, end effector 1902, and/orgripper 1904 without damaging high value and high down time components.Typical overload scenarios are automatically resettable by an additionalretention system that keeps the magnetic coupling in close proximity tothe robotic arm 102's phalange 1906. The compliant magnetic couplingmechanism also allows for quick replacement of end effector 1902 withinminutes.

FIG. 19D shows an example configuration of magnets 1908 that provide forthe magnetic coupling. With reference to FIG. 19D, the end effector 1902comprises a plurality of (e.g., 3) magnetic components 1908 arranged onthe proximal surface of the end effector 1902. The magnetic componentscan be glued or screwed onto the proximal surface. In the depictedexample, the three magnetic components 1908 are equally spacedcircumferentially (i.e., 60 degrees apart). Further, the distal surface1900 of the robotic arm phalange 1906 comprises another set of magneticcomponents (not depicted) that are arranged in the same configuration.As such, the magnetic components 1908 on the end effector are configuredto attract the magnetic components arranged on the distal surface 1900of the robotic arm phalange 1906, thus coupling the end effector and therobotic arm.

FIG. 19B show the compliant magnetic coupling mechanism in the coupledstate. In FIG. 19B, the proximal surface of the end effector 1902 isattached to the distal surface 1900 of the robotic arm via the magneticcomponents 1908 on the end effector and a corresponding set of magneticcomponents arranged on the surface 1900 (not depicted).

FIG. 19C shows the mechanism in the decoupled state with magnet 1908 andpins 1910 showing. When a force (e.g., a lateral force) above a certainthreshold is applied to any part of the system (e.g., the end effector1902), the system allows the end effector and the robotic arm totemporarily break away without completely detaching from each other,such that they can automatically reattach to each other later. Withreference to FIG. 19C, the end effector 1902 further comprises anelongated member 1903 extending through a hole in the proximal surfaceof the end effector. In the depicted example, the elongated member is ascrew. The proximal end of the elongated member can be affixed to thedistal surface 1900 of the motion device, for example, via a screwingmechanism. For example, the elongated member 1902 can be screwed into ascrew hole in the surface 1900 of the robotic arm when the end effectoris first attached to the robotic arm.

The elongated member comprises an end stopper piece 1905 configured toprevent the distal end of the elongated member 1903 from passing throughthe hole in the proximal surface of the end effector. In the depictedexample, the end stopper piece 1905 is a bolt, but it should beappreciated that any end stopper piece that cannot pass through the holein the proximal surface of the end effector can be used.

As shown in FIG. 19C, when a force above a certain threshold is appliedto the system, it can cause one or more of the magnetic components 1908to detach from one or more of the magnetic components on the surface1900 of the robotic arm. However, the elongated member 1903 remainaffixed to the surface 1900 and the end stopper piece 1905 prevent theend effector from completely detaching from the robotic arm.

When the force causing the breakaway is no longer present, the magnetcomponents 1908 cause the proximal surface of the end effector toautomatically attach to the distal surface of the motion device, thusautomatically resetting the connection and maintaining the previousalignment between the robotic arm and the end effector.

In some embodiments, one of the distal surface of the robotic arm andthe proximal surface of the end effector comprises one or more pins; andthe other one of the distal surface of the robotic arm and the proximalsurface of the end effector comprises one or more openings for receivingthe one or more pins. As shown in FIGS. 19C and 19D, proximal surface ofthe end effector comprises three pins 1910. The surface 1900 of therobotic arm comprises three openings or holes (not depicted) arranged inthe same configuration for accommodating the pins. The pins have taperedtops that make it easier to fit into the openings. The pins allow for amore precise and secure resetting.

In some embodiments, the proximal end of the rod 1907 (FIG. 19D) isattached to a flexible tube in the robotic arm. Thus, during thetemporary breakaway (FIG. 19C), the tube in the robotic arm can extendor stretch such that the connection between the rod 1907 and the tube inthe robotic arm remain secure.

In some embodiments, the compliant magnetic coupling mechanism comprisesa plurality of elongated members 1903 operating in a substantiallysimilar manner to ensure a precise and secure automatic resetting.

In some embodiments, the system comprises one or more sensors to detectthe occurrence of a breakaway (e.g., between the robotic arm phalangeand the end effector), and the degree of the breakaway. Any type ofsensors can be used, such as hall-effect sensors or inductance sensors.In some embodiments, the sensors are arranged on the motion device(e.g., the phalange), the end effector, or both, and are configured todetect detachment or increase of distances between components on themotion device and the components on the end effector.

For example, based on the signals from the sensors, the system candetermine that a magnetic component on the motion device is temporarilydetached from the corresponding magnetic component on the end effector.In accordance with a determination that a detachment has occurred, thesystem moves (e.g., retracts) the robotic arm to prevent furtherdetachment. In some embodiments, the system can determine the degree ofthe breakaway (e.g., one magnetic component detached, two magneticcomponents detached, all magnetic components detached, one or more pinsdetached, distance between the proximal surface of the end effector tothe end stopper piece) and provides an output accordingly. For example,the system can move the robotic arm to prevent further breakaway. Asanother example, the system can move the robotic arm away to allow forautomatic resetting. As another example, the system can issue an alertsuch that a human operator can inspect the environment and makeadjustments accordingly (e.g., to the packages, to the robotic system).

FIGS. 19E-G depict an exemplary magnetic coupling mechanism for roboticarm end effectors, in accordance with some embodiments. In FIG. 19E, arobotic arm tool 1920 comprises a rod 1922 and a tool base 1924. Thetool base 1924 is affixed to the distal end of the rod 1922. A magneticcoupling mechanism is used to couple a detachable tool 1926, such as asuction gripper, to the tool changer base 1922 of the robotic arm tool.Magnetic coupling between the robotic arm tool 1920 and the detachabletool 1926 allows for breakaway when a sufficient force is applied to therobotic arm tool 1920 and/or the detachable tool 1926 to separate thetwo. In some embodiments, the decoupling is achieved via a tool rack, asdiscussed in further details herein. Further, the magnetic couplingmechanism can be used for any coupling purposes, for example, forcoupling any motion device and any detachable end effector.

FIG. 19F depicts an exemplary tool changer base 1924, in accordance withsome embodiments. The tool changer base 1924 comprises an embeddedmagnet 1930 at the distal end. The embedded magnet can be affixed to therest of the tool changer base 1924 via, for example, glue or urethaneresin. In the depicted example, the cross-section of the embedded magnet1930 is of a ring shape. It should be appreciated that the cross-sectionof the embedded magnet can be of any shape, such as circle shape,ellipse shape, or polygon shape (e.g., rectangle, square). In someembodiments, the tool changer is compact and cylindrical in shape with adiameter of 35 mm to facilitate reaching into deep containers andgripping items near walls and corners. The tool changer can also be ofany shape including shapes that facilitate the coupling of detachabletools in a desirable orientation. As discussed above, the tool changerbase 1924 is affixed to the distal end of a rod. In some embodiments,the rod has a hollow center, and a hollow space is formed and extendslongitudinally through the rod and the tool changer base 1924.

FIG. 19G depicts an exemplary detachable tool 1926, in accordance withsome embodiments. The detachable tool 1926 comprises an embedded magnet1932 at the proximal end of the detachable tool. The embedded magnet1932 can be affixed to the rest of the detachable tool via, for example,glue or urethane resin. In the depicted example, the cross-section ofthe embedded magnet 1932 is of a ring shape and is identical orsubstantially identical to the cross-section of the embedded magnet1930.

The embedded magnet 1930 and the embedded magnet 1932 are arranged suchthat the distal end of the embedded magnet 1930 and the proximal end ofthe embedded magnet 1932 attract each other. In some embodiments, theembedded magnets are configured to produce a pull force above apredefined threshold. In some embodiments, the embedded magnets canattach to each other with 30 pound-force (lbf) of force or more toeffectively facilitate high speed tool changing and picking of items.

In operation, when the tool changer base 1924 and the detachable tool1926 are placed in proximity with each other, the embedded magnet 1930and the embedded magnet 1932 are coupled together via magnetic force. Insome embodiments, the embedded magnet 1930 and the embedded magnet 1932are configured to align in a manner such that the hollow space formedthrough rod 1922 continues to extend longitudinally into the detachabletool. Accordingly, the detachable tool 1926 is securely coupled to thedistal end of the robotic arm tool 1920 in an aligned manner.

The automatic alignment can be achieved, for example, by making surethat the cross-sections of the embedded magnets have identical orsubstantially identical shapes and, optionally, have identical orsubstantially identical dimensions. Because magnets have magnetic fieldsthat attract and these fields, when of similar strengths and sizesnaturally align with each other, such configuration of the embeddedmagnets would allow the magnets to automatically align and center. Forexample, cross-sections of embedded magnet 1930 and 1932 each include acircular hole, and the embedded magnets 1930 and 1932 can automaticallyalign in a manner such that the circular holes are concentric and/ormatch up.

The robotic arm tool 1920 and the detachable tool 1926 can comprise agripping mechanism for gripping objects. In the depicted example in FIG.19G, the detachable tool 1926 comprises a vacuum cup 1934 at the distalend. In some embodiments, the detachable tool comprises a longitudinalhollow center that extends longitudinally through the length of thedetachable tool. As such, when the detachable tool 1926 is coupled tothe distal end of the robotic arm tool 1920, a single hollow space isformed and extends longitudinally through the robotic arm tool 1920(including the rod 1922 and the tool changer base 1924) and thedetachable tool 1926. The hollow space can function as a vacuum chamber.In operation, when the detachable tool 1926, specifically, the vacuumcup 1934, is placed at a desirable location on the surface of an object,the robotic system activates vacuum pass-through such that the vacuumcup is coupled to the object and achieves gripping.

The robotic arm tool 1920 and the detachable tool 1926 can support avariety of gripping mechanisms configured to manipulate objects ofdifferent sizes, weights, and surface areas. For example, differentdetachable tools can include suction cups of different sizes, with thelarger suction cup configured to manipulate heavier and/or largerobjects. Further, the robotic arm tool and the detachable tool canaccommodate for pass-through of positive pressure pneumatics andelectrical connections, while maintaining minimal impact to cycle time.The tool changer can also be adapted to support actively actuatedgrippers, which can be controlled (e.g., opened and closed) viadifferent types of pass-through (e.g., pneumatics, electrical).

As depicted in FIG. 19G, the detachable tool 1926 comprises a groovesection 1928. The groove section 1928 is configured to interface with atool rack, which can retain the detachable tool 1926 in the z direction,as discussed below.

FIGS. 19H-J depict a tool rack for accommodating one or more detachabletools, in accordance with some embodiments. In FIG. 19H, a tool rack1940 is secured to a base via a number of (e.g., 3) bolts such that itremains stationary. The tool rack comprises one or more slots 1942 and1944. In the depicted example, the tool rack 1940 has two slots. Eachslot has a widened opening for receiving a detachable tool and a notchfor retaining the received detachable tool in the z direction.

As shown in FIG. 19I, the tool rack 1940 supports two detachable toolsvia the two slots. In the depicted example, the two detachable toolsinclude identical tool changer bases but include suction cups ofdifferent sizes, with the bigger suction cup configured to manipulateheavier/bigger objects. Although suction cups are depicted, it iscontemplated that other detachable tools can be retained by the toolrack in a similar manner. For example, detachable tools having activelyactuated grippers that mechanically open and close can also be receivedby the tool rack.

FIG. 19J depicts a detachable tool being received, via the widenedopening of the slot, into the notch of the slot. The notch interfaceswith the groove section of the detachable tool such that the detachabletool is retained in the z direction. Further, a magnetic force isgenerated between the embedded magnet of the detachable tool and thematerial of the rack (e.g., ferrous rack) such that the detachable toolis securely mounted.

The tool rack 1940 can facilitate the coupling between the robotic armtool and the detachable member. In operation, the robotic arm moves toplace the tool changer base in proximity to (e.g., over) a detachabletool mounted over the tool rack. The tool changer base is lowered in thez direction such that it makes contact with the detachable tool.According to the magnetic coupling mechanism described above, the toolchanger base and the proximal end of the detachable tool are coupledtogether in an aligned and centered manner. After coupling is achieved,the robotic arm moves the detachable tool out of the notch (e.g., via amovement on the x-y plane) and away from the tool rack. In someembodiments, after the robotic arm moves the detachable tool out of thenotch such that the groove section of the detachable tool no longerinterfaces with the notch, the robotic arm subsequently pulls upwards inthe z direction to move the detachable tool away from the tool rack.

The tool rack 1940 can further facilitate the decoupling between therobotic arm tool and a detachable tool coupled to the robotic arm tool.In operation, the robotic arm, which is coupled to the detachable tool,moves the detachable tool toward a widened opening of an available notch(e.g., via a movement on the x-y plane) such that the groove section ofthe detachable tool eventually interfaces with the notch. Accordingly,the detachable tool is retained by the tool rack via the notch in the zdirection. Further, an attraction force is generated between theembedded magnet of the detachable tool and the tool rack to furthersecure the retention. Subsequently, the robotic arm pulls upwards (e.g.,along the z direction), thus decoupling the robotic arm tool and thedetachable tool, which remains mounted by the tool rack.

The tool rack can be secured in any orientation or angle, and the motionpaths of the robotic arm can be programmed accordingly to achievecoupling and decoupling described above. For example, the tool rack canbe secured in an upright orientation, with the widened openings of theslots facing upward. Accordingly, the robotic arm can move downward(e.g., along the z direction) such that the groove section of adetachable tool interfaces with a slot and pull (e.g., via a movement onthe x-y plane) to decouple the detachable tool from the robotic arm. Itshould be appreciated that the motion paths of the robotic arm can becurved and/or angled based on the orientation of the tool rack.

In some embodiments, while the robotic arm tool is coupled with a firstdetachable tool, the system determines whether a second detachable toolis needed. In some embodiments, the determination is based oncharacteristics (e.g., surface, shape, size, weight) of the object to begripped, characteristics of the second detachable tool (e.g., size, typeof griping mechanism, location on the tool rack), characteristics of thefirst detachable tool, or any combination thereof. The system can makethe determination before attempting to grip the object, or make thedetermination after the system attempts to and fails to grip the object(e.g., due to the surface, shape, size, or weight of the object).

In accordance with a determination that the second detachable tool isneeded, the robotic arm tool decouples from the first detachable tool,for example, by mounting the first detachable tool onto an availableslot on a tool rack and pulling away from the tool rack to decouple thefirst detachable tool from the tool changer base. In some embodiments,the system stores an association between the first detachable tool andthe location of the slot that retains the first detachable tool.Further, the robotic arm tool subsequently couples the tool changer basewith the second detachable tool, for example, by picking up the seconddetachable tool from the tool rack.

In some embodiments, when a detachable tool is held in a tool rack, thesystem stores the corresponding location (e.g., corresponding slot ofthe tool rack) of the detachable tool. In some embodiments, the locationis pre-assigned to the detachable tool. Further, one or morepre-programmed motion path of the robotic arm can be stored inassociation with the detachable tool and/or the location on the toolrack. In operation, based on characteristics of a target object (e.g.,size, height, or shape), the system may determine an appropriatedetachable tool (e.g., the appropriately sized suction cup) and retrievethe corresponding location. The system may then execute a pre-programmedmotion (e.g., a up-side-down U path) to attach the detachable tool tothe robotic arm and effect the tool change.

The above-described magnetic coupling mechanism provides a flexiblemanipulation solution for manipulating objects of a wide array of sizes,weights, and surfaces, while minimizing impact to cycle time. In someembodiments, the mechanism allows for quick and secure tool exchangewithin 0.5-1 second (e.g., 0.5 second, 0.6 second, 0.7 second, 0.8second, 0.9 second, 1 second), compared to over 5 seconds for currentlyavailable methods.

Advantageously, the above-described magnetic coupling mechanism and toolrack can securely exchange tools at a speed equal to or faster than asingle item pick cycle without any manual intervention. As such,potential bottleneck caused by tool changes is reduced or eliminated.For example, a warehouse that handles a wide range of package sizes mayrequire frequent tool exchanges to allow the robotic arm to properlygrip the packages during picking and sorting. Frequent tool exchangesthat take over 5 seconds each time may unreasonably constrain thethroughput of the picking and sorting process. However, if the toolexchange takes equal to or less than the cycle time, then the pickingand sorting of different items can be streamlined more easily and beperformed more efficiently.

A further advantage of the magnetic coupling mechanism and the tool rackis that manual intervention is not necessary. The magnetic couplingmechanism is self-aligning once the tool changer base is in closeproximity to the detachable tool. The self-aligning nature of themagnetic coupling mechanism obviates the need for manual intervention,which is both labor and time intensive, as compared to the automatedsystem described herein. A human worker takes longer to changedetachable tools, during which time packages are not being moved orsorted.

A further advantage of the magnetic coupling mechanism is thatmechanical coupling methods are not necessary to achieve gripping and/ormaintain contact. Mechanical coupling methods require additionalcomplexity such as moving parts and electrical wiring, which increasescosts as compared to the magnetic coupling mechanism. Furthermore,mechanical systems introduce additional failure points through movingparts. Mechanical coupling methods also require additional time toproperly couple (or decouple) the detachable tool to the tool changerbase. Specifically, additional contact needs to be established betweenthe detachable tool and the tool changer base (e.g., latches, electricalwiring, clamps), addition motion or maneuvering of the robotic arm isneeded, and complex actuation mechanisms may be required. Althoughmechanical coupling methods are unnecessary, it is contemplated thatmechanical coupling methods can be used in addition to the magneticcoupling mechanism if additional security is needed. For example, somepackages may weigh more than the pull force that the magnetic couplingmechanism is capable of, and mechanical coupling mechanisms may beadditionally deployed. In another example, the magnetic couplingmechanism can facilitate self-aligning for additional mechanicalcoupling mechanisms.

A further advantage of the magnetic coupling mechanism is that therobotic arm tool and the detachable tool have a compact form-factor. Forexample, the tool changer base can have a diameter of 35 millimeters. Acompact form-factor is desirable because the robotic arm tool can picksmall objects out of a deep container. A compact form-factor is furtherbeneficial when gripping items near walls and corners. Since warehouseshandle a wide variety of package sizes, the ability to flexibly maneuverthe robotic arm tool to facilitate picking and sorting is desirable.

FIG. 20 depicts end effector 2000 with loose bag cup gripper 2002, whichis designed to increase the ability to pick, hold, and manipulate itemslike a loose polybag 2004. Loose bag cup gripper 2002 sucks the bag 2004into the cup. The increased surface area provides for secure holdingwithout a large volume of air flow. As compared to the suction nozzlegripper described above with respect to FIG. 6A, loose bag cup gripper2002 is optionally more rigid, deeper, and/or wider.

FIGS. 21A-E depict an exemplary loose bag cup gripper for robotic armend effectors, in accordance with some embodiments. In FIG. 21C, theloose bag cup gripper 2002 comprises a proximal end 2102 and a distalend 2104. The proximal end 2102 can be attached to the end effector viaa magnetic attachment, similar to embodiments described above. Theproximal end 2102 can also be attached to the end effector via othermeans; for example, via mechanical means (e.g., latches) or via threadedconnections (e.g., screw-on).

FIG. 21B depicts a cross-sectional view of loose bag cup gripper 2002.Loose bag cup gripper 2002 comprises a primary chamber 2106 andsecondary chamber 2108. In some embodiments, secondary chamber 2108 isexternal to primary chamber 2106 and surrounds primary chamber 2108. Inthe depicted example, the primary chamber 2106 and the secondary chamber2108 are both cylindrical, wherein the primary chamber 2106 has asmaller diameter than the secondary chamber 2108. Secondary chamber 2108optionally surrounds and is concentric with primary chamber 2106. Thisconfiguration can be achieved, for example, by placing a first hollowcylinder inside a second hollow cylinder, thus forming a primary chamber(i.e., the first hollow cylinder) and a secondary chamber (i.e., thespace between the first hollow cylinder and the second hollow cylinder).

In some embodiments, secondary chamber 2108 is connected to primarychamber 2106 via one or more connections 2110 (e.g., holes) such thatair can flow between the primary chamber 2106 and the secondary chamber2108. In the depicted example, secondary chamber 2108 connects toprimary chamber 2106 at four levels, with five connections 2110 at eachlevel. However, it should be noted that any connection arrangement canbe used. In one embodiment, connection 2110 is 4 millimeters wide and 20millimeters long.

In some embodiments, the primary chamber 2106 comprises an opening atthe proximal end 2102 and an opening at the distal end 2104. In thedepicted example, the cross-section of the opening at the proximal end2102, the cross-section of the opening at the distal end 2104, and thecross-section of the primary chamber 2106 are all of a ring shape. Itshould be appreciated that the cross-sections of the opening at theproximal end 2102, the opening at the distal end 2104, and the primarychamber 2106 can be of any shape, such as ellipse shape or polygon shape(e.g., rectangle, square). The cross-sections of the opening at theproximal end 2102, the opening at the distal end 2104, and the primarychamber 2106 can optionally be the same shape. Alternatively, one ormore of the cross-sections can be a different shape than the othercross-sections. In one embodiment, the opening at the distal end 2104 issmall enough so that only one bag 2004 is gripped at one time. In someembodiments, the loose bag cup gripper 2002 has an outer diameter of 50millimeters and a length of 110 millimeters. The dimensions of loose bagcup gripper 2002 can also be scaled to work with bags of differentthicknesses and different weights.

The depicted embodiment shows that the opening at proximal end 2102 hasa diameter that is smaller than a diameter of the primary chamber 2106.The depicted embodiment also shows that the opening at the distal end2104 is shaped as a curved funnel. The opening at the distal end 2104may also be shaped as other shapes, for example a cone.

As shown in FIG. 21B, the opening at the proximal end 2102 of the loosebag cup gripper 2002 can be configured to align with the distal end of arobotic arm to create a single hollow space that forms and extendslongitudinally through the rod 2006 and the loose bag cup gripper 2002,specifically, primary chamber 2106 and secondary chamber 2108. Thehollow space can function as a vacuum chamber. In operation, when theloose cup gripper 2002 is placed at a desirable location in proximity toa loose bag 2004, the robotic system activates a vacuum pass-throughsuch that the loose bag cup gripper 2002 grips the loose bag 2004.

FIGS. 22A-B depict loose bag cup gripper 2002 in operation. In FIG. 22A,a vacuum flow is created through the primary chamber 2106 of loose bagcup gripper 2002 (e.g., via an air flow source) and the bag 2004 isdrawn upward into the primary chamber 2106 through the opening at thedistal end 2104. After the bag 2004 is drawn upward and embedded in theprimary chamber 2106, the bag 2004 obstructs and arrests the vacuumflow. In some embodiments, primary chamber 2106 is long enough such thatits geometry prevents bag 2004 from being vacuumed completely into theprimary chamber 2106. One or more connections 2110 therefore remainsunblocked by bag 2004, thereby maintaining the air flow connectionbetween primary chamber 2106 and secondary chamber 2108. The arrestedvacuum flow creates a pressure differential between the inside of bag2004 and hollow chambers 2106, 2108, thereby gripping bag 2004vertically and laterally.

In one embodiment, gripper 2002 is made from slippery plastic such asDelrin or PTFE, although other materials may be used. In anotherembodiment, gripper 2002 is made from slippery cast urethane resins. Thevacuum force for primary chamber 2106 may be 10 pounds-force axially,and the vacuum force for secondary chamber 2108 may be 16 pounds-forcelaterally.

In the depicted example, loose bag 2004 is composed of plastic and isdeformable. The deformability allows a portion of the bag 2004 to bedrawn into primary chamber 2106. Once a portion of the bag 2004 is drawninto primary chamber 2106, the pressure differential between the insideof the bag 2004 (i.e., pressure of the atmosphere) and the pressure ofthe primary/secondary chambers causes the bag to adhere to the lateralwall of the primary chamber. Once the robotic arm tool has repositioneditself to a desired drop-off location, the vacuum flow can be turnedoff, allowing the bag 2004 to exit through distal end 2104 of the bag'sown weight. Although a plastic bag is depicted, it is contemplated thatloose bag cup gripper 2002 can be used to grip other deformablematerials, like paper or fabric.

In some embodiments, the system will identify regions on bag 2004 thatare likely to contain a label (e.g., barcode) in accordance with theembodiments described in FIGS. 10A-C. The system may position gripper2002 with sufficient distance from the regions that are likely tocontain a label such that the label is not obscured when the bag 2004 isgripped by gripper 2004.

Applying a lateral suction force has the advantage of making the bag2004 resistant to pullout from both gravitational and side-loadedforces. The loose bag cup gripper 2002 can securely hold a polybagduring high acceleration or during high loading scenarios. The presentembodiment further has the advantage of reducing the need to employmultiple vacuum cups in an array to prevent side-loaded pullout.Multiple vacuum cup arrays are bulky, complex, costly, and still do notfully mitigate side-loaded detachment of gripped polybags. The presentembodiment further has the advantage of reducing the need to employparallel actuation methods, like mechanical grippers that can be openedand closed. Parallel actuation methods also increase the complexity andcost of the actuation system. Although depicted embodiments reduce theneed to employ additional actuators and/or additional actuation methods,additional actuators and/or actuation methods can nevertheless be usedif additional gripping security is desired. The present embodimentfurther has the advantage of being able to securely grip any part ofreadily available polybags. Specialized bags with specific suctionregions are not required, reducing the complexity of the robotic armsystem and the cost of the employed bags.

In an alternate embodiment, loose bag cup gripper 2002 can be modifiedto include a suction cup located at the distal end 2104. The suction cupis optionally less rigid than loose bag cup gripper 2002, therebyallowing the suction cup to better grip rigid surfaces. The suction cupis coupled to the loose bag cup gripper 2002 such that it continues thevacuum pass-through comprising the robotic arm tool 1920 and the loosebag cup gripper 2002. In this embodiment, the same robotic arm endeffector attachment can be used to grip both polybags and items withrigid surfaces, like boxes.

In operation, the modified loose bag cup gripper 2002 is placed at adesirable location in proximity to a target item. The robotic systemthen activates a vacuum pass-through such that the modified loose bagcup gripper 2002 applies a vertical suction force to the target item. Ifthe target item is a polybag, the target item will be at least partiallydrawn into primary chamber 2106 and also held laterally by secondarychamber 2108, in accordance with embodiments described in FIGS. 22A-B.If the target item is rigid, the target item will be suctioned to thesuction cup, thereby achieving gripping.

The system is able to switch the type of end effector (e.g., switchingbetween the gripper 1904 and gripper 2002) during the operationautomatically based on the object characteristics (e.g. dimension,weight, surface material) in the tote. There are multiple types of endeffectors (e.g., multiple types of cups) that are placed on a fixturenearby the robot. When the robot needs to switch the end effector, itwill conduct a certain motion so that the current engaged end effectorwill be left on the fixture and then engage another cup on the fixture.In some cases, the end effectors are attached with magnetic force orvacuum force.

FIG. 23 depicts a package orientation system 2300 for orienting package2302 so that label 2304 is detectable by scanner 2306, in accordancewith some embodiments. Detecting label 2304 has the advantage ofconfirming that the package's orientation is suitable for label readingat a later step. For example, if scanner 2306 is located directly aboveupper conveyor belt 2308, scanner 2306 will preferably only detect label2304 when package 2302 is oriented with its label 2304 facing upwards.Orienting packages with labels facing upwards is advantageous becausemany existing warehouses have pre-existing infrastructure built to readlabels that are facing up. Since packages are often oriented in otherorientations during transit and picking, it is desirable to create acompact, quick, and cost-efficient system to re-orient packages so thattheir labels face a desirable direction (e.g., upwards).

Package orientation system 2300 comprises a scanner 2306, an upperconveyor belt 2308, a flipping conveyor belt 2310, and a lower conveyorbelt 2312. In some embodiments, flipping conveyor belt 2310 is locatedat a proximal end of upper conveyor belt 2308. In a preferredembodiment, flipping conveyor belt 2310 is further located such that,when it is in a first orientation (e.g., upright orientation), package2302 travelling on upper conveyor belt 2308 runs into and cannotcontinue past flipping conveyor belt 2310. As discussed in detail below,the flipping conveyor belt 2310 is an angled conveyor to facilitatereorient the object via controlled tumbling manipulation. The firstorientation of the flipping conveyor belt is tuned to optimize thesuccess rate in reorienting objects. In some embodiments, the firstorientation of the flipping conveyor belt is 90 degrees (i.e.,perpendicular to the upper conveyor belt) or substantially 90 degrees(e.g., 80 degrees-100 degrees).

Lower conveyor belt 2312 is preferably located below upper conveyor belt2308 and flipping conveyor belt 2310. In some embodiments, C-chute 2314is located at a distal end of upper conveyor belt 2308.

Scanner 2306 is preferably located at or near the point where package2302 cannot continue past flipping conveyor belt 2310. In oneembodiment, scanner 2306 is located directly above upper conveyor belt2308 and points down at upper conveyor belt 2308. However, it should beappreciated that scanner 2306 can be placed in other positions relativeto upper conveyor belt 2308 as well. Scanner 2306 can be configured torecognize the label 2304 on package 2302. Label 2304 can comprise anymethod of storing information about a package. For example, label 2304can be a 1D/2D barcode, fiducial marker, a QR code, or any combinationthereof. Label 2304 can be machine-readable or readable by humans. Inone embodiment, scanner 2306 continuously attempts to detect a label.

FIGS. 24A-D depict package orientation system 2300 in operation. Package2302 is placed on upper conveyor belt 2308 by any method of transportingpackages. For example, package 2302 can be placed onto belt 2308 by arobot, a human, or a different conveyor belt. In some embodiments, upperconveyor belt 2308 is already running when package 2302 is placed onupper conveyor belt 2308. Alternatively, upper conveyor belt 2308 canbegin to run when package 2302 is placed onto the belt 2308. In FIG.24A, package 2302 is placed in a first orientation such that label 2304is not recognizable by scanner 2306, so upper conveyor belt 2308continues to run, moving package 2302 along belt 2308 and towardsflipping conveyor belt 2310.

If the package is placed onto the conveyor belt 2308 with the bar codeon its two lateral side, flipping the package would not make it visibleto the scanner 2306. Thus, in some embodiments, before placing thepackage 2302 onto the conveyor belt 2308, the system makes sure that thebarcode is not on the two lateral surfaces of the package. In someembodiments, the package is picked up using a robotic arm and rotatedcontinuously (e.g., along the z-axis) while a second scanner scans thepackage. Based on the image captured by the second scanner, the systemdetermines whether the package has been rotated such that the surfacewith the barcode on is one of the four sides that would be flipped whenthe package is placed onto the conveyor belt, rather than the twolateral sides. In accordance with a determination that the surface isone of the four sides that would be flipped, the robotic arm places thepackage onto the conveyor belt 2308. In accordance with a determinationthat the surface with the barcode on is not one of the four sides, therobotic arm continues to rotate the package around the z-axis (e.g., byrotating the gripper).

In FIG. 24B, package 2302 moves along upper conveyor belt 2308 and comesinto contact with flipping conveyor belt 2310. Flipping conveyor belt2310 preferably runs in a direction that will pull package 2302 upwardsand away from upper conveyor belt 2308 so that package 2302 tumbles intoa second orientation. Flipping conveyor belt 2310 can be continuouslyrunning. Alternatively, flipping conveyor belt 2310 can begin runningwhen scanner 2306 detects package 2302, but does not detect label 2304.If label 2304 is still not recognizable by scanner 2306 in the secondorientation, belts 2308 and 2310 will continue to run and package 2302will continue to tumble into different orientations.

In some embodiments, one or both of the upper conveyor belt 2308 andflipping conveyor belt 2310 are cleated to help grip package 2302 and tofacilitate tumbling, pivoting, or flipping package 2302 into a secondorientation (e.g., by 90, 180, 270 degrees). In one embodiment, theupper conveyor belt 2308 employs a cleated conveyor belt with lowfriction such that scanner 2306 has time to detect label 2304 beforepackage 2302 tumbles into the second orientation. In another embodiment,the flipping conveyor belt 2310 employs a conveyor belt with highfriction to facilitate tumbling package 2302. It is contemplated thatany combination of cleated conveyor belts, high friction belts, and lowfriction belts may be used.

In FIG. 24C, the system determines that the package 2302 has tumbledinto an orientation such that it is ready to be processed by adownstream sorter. For example, if the downstream sorter has a singlebarcode reader that requires the barcode label of the object facingupwards, the system would determine that the package is in theappropriate orientation when the scanner 2306 recognizes a barcode onthe top surface of the package. As another example, if the downstreamsorter has a 5 sided scanning tunnel, the system would determine thatthe package is in the appropriate orientation when the bar code is noton the bottom surface of the package (e.g., based on an output of ascanner scanning the bottom surface). Once the system determines thatthe package is in the appropriate orientation, upper conveyor belt 2308and flipping conveyor belt 2310 stop running so that package 2302 stopstumbling.

In FIG. 24D, flipping conveyor belt 2310 moves so that package 2302 hasan unobstructed path to continue along upper conveyor belt 2308. In apreferred embodiment, flipping conveyor belt 2310 moves to a secondorientation (e.g., swings open), creating sufficient space betweenflipping conveyor belt 2310 and upper conveyor belt 2308 for package2302 to pass through. It should be appreciated that flipping conveyorbelt 2310 can move via other methods as well. For example, flippingconveyor belt 2310 can slide upwards to create sufficient space betweenflipping conveyor belt 2310 and upper conveyor belt 2308 for package2302 to pass through, as shown in FIG. 24E.

In some embodiments, when a second package is placed onto the conveyorbelt 2308, the flipping conveyor belt 2310 moves to the firstorientation (e.g., upright as shown in FIG. 24A) or position (lower asshown in FIG. 24F) in response to a determination that the secondpackage on the conveyor belt 2308 needs to be flipped based on the imagedata captured by the scanner 2306. In some embodiments, the flippingconveyor belt 2310 moves to the first orientation (e.g., upright asshown in FIG. 24A) or position (lower as shown in FIG. 24F) after thefirst package passes through.

In some embodiments, the scanner 2306 continues scanning the packagewhile the package is being flipped. The scanner 2306 captures one ormore images and determines, based on the one or more images, whether abar code has been successfully detected and read. In accordance with adetermination that a bar code has been successfully read, the systemlets the package to move along, for example, by moving the flippingconveyor belt (e.g., rotate or moving upwards). In accordance with adetermination that a bar code has not been successfully read, the systemcontinues flipping the package. Accordingly, a downstream scanner is notneeded.

FIGS. 25A-B depict package orientation system 2300 handling flat package2502. In a preferred embodiment, package orientation system 2300comprises C-chute 2314. C-chute 2314 is preferably located at a distalend of upper conveyor belt 2308 and configured to receive packagestraveling along upper conveyor belt 2308. C-chute 2314 preferably wrapsaround the distal end of upper conveyor belt 2308 such that flat package2502 leaving the distal end of belt 2308 will be flipped over. Flatpackage 2502 can have substantially larger measurements along twodimensions and relatively small measurements along a third dimension.Flat package 2502 can be, for example, an envelope or a polybag. Flatpackage 2502 can also be characterized as not having enough room alongthe shorter third dimension to affix label 2504 to. As such, label 2504will preferably be on one of only two sides of flat package 2502.

In operation, flat package 2502 is placed on upper conveyor belt 2308 byany method of transporting packages. For example, flat package 2502 canbe placed onto belt 2308 by a robot, a human, or a different conveyorbelt. Flat package 2502 will preferably initially proceed on upperconveyor belt 2308 towards the proximal end and flipping conveyor belt2310. If scanner 2306 fails to detect label 2504 when flat package 2502arrives at the proximal end of belt 2308, package orientation system2300 will determine a height of flat package 2502. Alternatively, theheight of flat package 2502 may be determined at another stage of thepicking or singulation process. In one embodiment, the height isdetermined using an RGB-D camera, although other methods for determiningheight may be used. In another embodiment, the height is determinedusing optical sensors, such as photoeyes. If the package is determinedto be a normal package, the package will be handled in accordance withsome embodiments shown in FIGS. 24A-D. If the package is determined tobe a flat package 2502, upper conveyor belt 2308 will reverse directionand send flat package 2308 to the distal end of belt 2308 and towardsC-chute 2314. Preferably, flat package 2308 will proceed past the distalend of belt 2308 and fall into C-chute 2314, which flips flat package2502 as flat package 2502 slides off C-chute 2314 onto lower conveyorbelt 2312. Label 2504 on flat package 2502 is now preferably visible andupwards facing.

In some embodiments, when the package passes through the flipperconveyor belt 2310, a conveyor belt 2309 receives the package. Withreference to FIG. 25C and FIG. 25D, the system determines whether thepackage is coming from the conveyor belt 2308 or the conveyor belt 2312.Based on the determination, the system changes the position and/ororientation of the conveyor belt 2309 such that the package can bereceived. In some embodiments, the determination can be made when thesystem determines whether the package is a flat package and needs to beflipped using the C-chute 2314 (FIG. 25A).

One advantage of package orientation system 2300 is that the system islow-cost because the system operates with a minimum amount of sensingand actuation. A further advantage of the system is that it is quick dueto automation. A further advantage of the system is that it is compact.Warehouse space can be valuable; it is therefore desirable that thespace required to process inventory is minimized. A further advantage ofthe system is that it is compatible with existing warehouseinfrastructure that requires labels to be upward-facing and has alimited amount of available floor space.

It is understood that the specific order or hierarchy of blocks in theprocesses/flowcharts disclosed is an illustration of exemplaryapproaches. Based upon design preferences, it is understood that thespecific order or hierarchy of blocks in the processes/flowcharts can berearranged. Further, some blocks can be combined or omitted. Theaccompanying method claims present elements of the various blocks in asample order, and are not meant to be limited to the specific order orhierarchy presented.

The previous description is provided to enable any person skilled in theart to practice the various examples described herein. Variousmodifications to these examples will be readily apparent to thoseskilled in the art, and the generic principles defined herein can beapplied to other examples. Thus, the claims are not intended to belimited to the examples shown herein, but are to be accorded the fullscope consistent with the language of the claims, wherein reference toan element in the singular is not intended to mean “one and only one”unless specifically so stated, but rather “one or more.” The word“exemplary” is used herein to mean “serving as an example, instance, orillustration.” Any example described herein as “exemplary” is notnecessarily to be construed as preferred or advantageous over otherexamples. Unless specifically stated otherwise, the term “some” refersto one or more. Combinations such as “at least one of A, B, or C,” “oneor more of A, B, or C,” “at least one of A, B, and C,” “one or more ofA, B, and C,” and “A, B, C, or any combination thereof” include anycombination of A, B, and/or C, and can include multiples of A, multiplesof B, or multiples of C. Specifically, combinations such as “at leastone of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B,and C,” “one or more of A, B, and C,” and “A, B, C, or any combinationthereof” can be A only, B only, C only, A and B, A and C, B and C, or Aand B and C, where any such combinations can contain one or more memberor members of A, B, or C. All structural and functional equivalents tothe elements of the various examples described throughout thisdisclosure that are known or later come to be known to those of ordinaryskill in the art are expressly incorporated herein by reference and areintended to be encompassed by the claims. Moreover, nothing disclosedherein is intended to be dedicated to the public regardless of whethersuch disclosure is explicitly recited in the claims. The words “module,”“mechanism,” “element,” “device,” and the like cannot be a substitutefor the word “means.” As such, no claim element is to be construed under35 U.S.C § 112(f) unless the element is expressly recited using thephrase “means for.”

What is claimed is:
 1. A system for orienting an object, comprising: a scanner configured to detect a label on the object; an upper conveyor belt; and a flipping conveyor belt located at an end of the upper conveyor belt, wherein the upper conveyor belt is configured to transport the object toward the flipping conveyor belt, wherein the flipping conveyor belt is configured to, in a first orientation, rotate and exert a frictional force on the object to reorient the object while the object is in contact with the upper conveyor belt, wherein the flipping conveyor belt is configured to, in a second orientation, allow the object to drop off the end of the upper conveyor belt, and wherein the flipping conveyor belt is configured to move from the first orientation to the second orientation based on an output of the scanner.
 2. The system of claim 1, wherein the end of the upper conveyor belt is a first end, wherein the system further comprises a curved chute at a second end of the upper conveyor belt, wherein the upper conveyor belt is configured to transport the object toward the curved chute, and wherein the curved chute is configured to rotate the object by 180 degrees.
 3. The system of claim 2, further comprising a lower conveyor belt configured to receive the object after the object is rotated by the curved chute.
 4. The system of claim 1, wherein the flipping conveyor belt is angled to the upper conveyor belt in the first orientation.
 5. The system of claim 1, wherein the upper conveyor belt is cleated.
 6. The system of claim 1, wherein the flipping conveyor belt is configured to pull a portion of the object upwards and away from the upper conveyor belt in the first orientation.
 7. The system of claim 1, wherein the flipping conveyor belt is configured to swing from the first orientation to the second orientation.
 8. The system of claim 1, wherein the scanner is configured to scan one or more surfaces of the object.
 9. The system of claim 1, wherein the object is a deformable bag.
 10. The system of claim 1, wherein the object is a box.
 11. A method for orienting an object, comprising: causing an upper conveyor belt to move the object toward a flipping conveyor belt located at an end of the upper conveyor belt; determining, based on an output from a scanner, whether the object is in one of one or more predefined orientations; in accordance with a determination that the object is not in one of the one or more predefined orientations, causing the flipping conveyor belt and the upper conveyor belt to run simultaneously to reorient the object while the flipping conveyor belt is in a first orientation; and in accordance with a determination that the object is in one of the one or more predefined orientations, causing the flipping conveyor belt to move to a second orientation such that the object is dropped off the end of the upper conveyor belt.
 12. The method of claim 11, wherein the end of the upper conveyor belt is a first end, the method further comprising: in accordance with a determination that the object is not in one of the one or more predefined orientations, determining, based on an output from an optical sensor, a height of the object; in accordance with a determination that the height of the object is below a certain threshold, causing the upper conveyor belt to move the object toward a second end; and in accordance with a determination that the height of the object is above the certain threshold, forgoing causing the upper conveyor belt to move the object toward the second end.
 13. The method of claim 11, wherein the upper conveyor belt is cleated.
 14. The method of claim 11, wherein the flipping conveyor belt is configured to pull a portion of the object upwards and away from the upper conveyor belt.
 15. The method of claim 11, wherein the determining whether the object is in one of the one or more predefined orientations is based on a configuration of a downstream sorter.
 16. The method of claim 11, wherein the determining whether the object is in one of the one or more predefined orientations comprises: scanning, using the scanner, a surface of the object to obtain image data; and determining, based on the image data, whether the image data includes information related to the object.
 17. The method of claim 16, further comprising: in accordance with a determination that the image data includes information related to the object, determining that the object is in one of the one or more predefined orientations; and in accordance with a determination that the image data does not include information related to the object, determining that the object is not in one of the one or more predefined orientations.
 18. The method of claim 16, wherein the information related to the object comprises a barcode.
 19. The method of claim 11, further comprising: in accordance with a determination that the object is not in one of the one or more predefined orientations, determining whether a height of the object exceeds a threshold; in accordance with a determination that the height of the object exceeds the threshold, causing the flipping conveyor belt and the upper conveyor belt to run simultaneously to reorient the object while the flipping conveyor belt is in the first orientation; and in accordance with a determination that the height of the object does not exceed the threshold, reversing movement of the upper conveyor belt to transport the object to a curved chute.
 20. A method for orienting an object, comprising: causing an upper conveyor belt to move the object toward a flipping conveyor belt located at an end of the upper conveyor belt; determining, based on an output from a scanner, whether a code on the object is read; in accordance with a determination that the code on the object is not read, causing the flipping conveyor belt and the upper conveyor belt to run simultaneously to reorient the object while the flipping conveyor belt is in a first orientation; and in accordance with a determination that the code on the object is read, causing the flipping conveyor belt to move to a second orientation such that the object is dropped off the end of the upper conveyor belt. 